Original: RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks
Revised: RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks

Original: Abstract
Revised: Abstract

Original: Recurrent neural networks (RNNs) are widely used throughout neuroscience asmodels of local neural activity.
Revised: Recurrent neural networks (RNNs) are widely used throughout neuroscience asmodels of local neural activity.

Original: Many properties of single RNNs are well characterized theoretically, but experimental neuroscience has moved in the direction ofstudying multiple interacting areas , and RNN theory needs to be likewise extended.
Revised: Many properties of single RNNs are well characterized theoretically, but experimental neuroscience has moved in the direction ofstudying multiple interacting areas , and RNN theory needs to be likewise extended.

Original: We take a constructive approach towards this problem, leveraging tools from nonlinear control theory and machine learning to characterize when combinations ofstable RNNs will themselves be stable.
Revised: We take a constructive approach towards this problem, leveraging tools from nonlinear control theory and machine learning to characterize when combinations ofstable RNNs will themselves be stable.

Original: Importantly, we derive conditions which allow for massive feedback connections between interacting RNNs.
Revised: Importantly, we derive conditions which allow for massive feedback connections between interacting RNNs.

Original: We parameterizethese conditions for easy optimization using gradient-based techniques and showthat stability-constrained ‘network of networks’ can perform well on challengingsequential-processing benchmark tasks.
Revised: We parameterizethese conditions for easy optimization using gradient-based techniques and showthat stability-constrained ‘network of networks’ can perform well on challengingsequential-processing benchmark tasks.

Original: Altogether, our results provide a principledapproach towards understanding distributed, modular function in the brain
Revised: Altogether, our results provide a principledapproach towards understanding distributed, modular function in the brain

Original:  Introduction
Revised:  Introduction

Original: The combination and reuse of primitive “modules" has enabled a great deal of progress in computerscience, engineering, and biology.
Revised: The combination and reuse of primitive “modules" has enabled a great deal of progress in computerscience, engineering, and biology.

Original: Modularity is particularly apparent in the structure of the brain,as different parts are specialized for different functions [Kandel et al., 2000].
Revised: Modularity is particularly apparent in the structure of the brain,as different parts are specialized for different functions [Kandel et al., 2000].

Original: Accordingly, mostexperimental studies throughout the history of neuroscience have focused on a single brain area inassociation with a single behavior [Abbott and Svoboda, 2020].
Revised: Accordingly, mostexperimental studies throughout the history of neuroscience have focused on a single brain area inassociation with a single behavior [Abbott and Svoboda, 2020].

Original: Similarly, RNN models of brainfunction have mostly been limited to a single RNN modeling a single area.
Revised: Similarly, RNN models of brainfunction have mostly been limited to a single RNN modeling a single area.

Original: However, neuroscienceis entering an age where recording from many different brain areas simultaneously during complexbehaviors is possible.
Revised: However, neuroscienceis entering an age where recording from many different brain areas simultaneously during complexbehaviors is possible.

Original: As experimental neuroscience has shifted towards multi-area recordings,computational techniques for analyzing, modeling, and interpreting these multi-area recordingshave blossomed [Abbott and Svoboda, 2020, Perich et al., 2021, Semedo et al., 2019, Yang and
Revised: As experimental neuroscience has shifted towards multi-area recordings,computational techniques for analyzing, modeling, and interpreting these multi-area recordingshave blossomed [Abbott and Svoboda, 2020, Perich et al., 2021, Semedo et al., 2019, Yang and

Original: Molano-Mazón, 2021].
Revised: Molano-Mazón, 2021].

Original: Despite this, RNN theory has lagged behind.
Revised: Despite this, RNN theory has lagged behind.

Original: The theoretical question of RNN stability is crucial for understanding information propagation andmanipulation [Vogt et al., 2020,
Revised: The theoretical question of RNN stability is crucial for understanding information propagation andmanipulation [Vogt et al., 2020,

Original: Engelken et al., 2020, Kozachkov et al., 2022].
Revised: Engelken et al., 2020, Kozachkov et al., 2022].

Original: The conditions underwhich single, autonomous RNNs are chaotic or stable are well-studied, in particular when the RNNweights are randomly chosen and the number of neurons tends to inﬁnity
Revised: The conditions underwhich single, autonomous RNNs are chaotic or stable are well-studied, in particular when the RNNweights are randomly chosen and the number of neurons tends to inﬁnity

Original: [Sompolinsky et al., 1988,
Revised: [Sompolinsky et al., 1988,

Original: Engelken et al., 2020].
Revised: Engelken et al., 2020].

Original: However, there is very little work addressing the theoretical question ofstability in ‘networks of networks’.
Revised: However, there is very little work addressing the theoretical question ofstability in ‘networks of networks’.

Original: Two facts make this question challenging.
Revised: Two facts make this question challenging.

Original: Firstly, connectingtwo stable systems does not, in general, lead to a stable overall system.
Revised: Firstly, connectingtwo stable systems does not, in general, lead to a stable overall system.

Original: Secondly, there is a massiveamount of feedback between brain areas, so one cannot reasonably assume near-decomposability[Simon, 1962, Abbott and Svoboda, 2020].
Revised: Secondly, there is a massiveamount of feedback between brain areas, so one cannot reasonably assume near-decomposability[Simon, 1962, Abbott and Svoboda, 2020].

Original: Given that the brain seems to dynamically reorganize and adapt interareal connectivity to meet taskdemands and environmental constraints [Miller and Cohen, 2001], this question of how stability ismaintained is of the utmost importance.
Revised: Given that the brain seems to dynamically reorganize and adapt interareal connectivity to meet taskdemands and environmental constraints [Miller and Cohen, 2001], this question of how stability ismaintained is of the utmost importance.

Original: Here we take a bottom-up approach, more speciﬁcally asking“what stability properties of the individual modules lend themselves to rapid reorganization?"
Revised: Here we take a bottom-up approach, more speciﬁcally asking“what stability properties of the individual modules lend themselves to rapid reorganization?"

Original: We focus on a special type of stability, known as contractive stability [Lohmiller and Slotine, 1998].
Revised: We focus on a special type of stability, known as contractive stability [Lohmiller and Slotine, 1998].

Original: Loosely, a contracting system is a dynamical system whose asymptotic behavior is exponentiallyindependent of its initial conditions.

Original: Contractive stability is a strong form of dynamical stability whichimplies many other forms of stability, such as input-to-state stability Sontag [2010].

Original: Contractionanalysis has found wide application in nonlinear control theory
Revised: Contraction analysis has found wide application in nonlinear control theory

Original: [Manchester and Slotine, 2017],synchronization [Pham and Slotine, 2007], and robotics [Chung and Slotine, 2009], but has onlyrecently begun to ﬁnd application in neuroscience and machine learning [Wensing and Slotine, 2020, Kozachkov et al., 2020, Revay and Manchester, 2020, Kozachkov et al., 2022, Centorrino et al., 2022,
Revised: [Manchester and Slotine,2017], synchronization [Pham and Slotine, 2007], and robotics [Chung and Slotine, 2009], but hasonly recently begun to ﬁnd application in neuroscience and machine learning [Wensing and Slotine,2020, Kozachkov et al., 2020, Revay and Manchester, 2020, Kozachkov et al., 2022, Centorrino et al.

Original:  Burghi et al., 2022].
Revised: ,2022, Burghi et al., 2022].

Revised: Loosely, a contracting system is a dynamical system whose asymptoticbehavior is exponentially independent of its initial conditions.

Original: Contraction analysis is particularly useful for neuroscience because it is directlyapplicable to non-autonomous systems, and it allows for modular contraction-preserving combinationproperties to be derived (Figure 1)
Revised: Contraction analysis is particularlyuseful for neuroscience because it is directly applicable to non-autonomous systems, and it allowsfor modular contraction-preserving combination properties to be derived (Figure 1)

Original: [Slotine and Lohmiller, 2001, Slotine, 2003].
Revised: [Slotine and Lohmiller, 2001, Slotine, 2003].

Original: See Section A1.2for a mathematical primer on contraction analysis.
Revised: See Section A1.2 for a mathematical primer on contraction analysis.

Original: Our contributions are three-fold:
Revised: Our contributions are three-fold:

Original: • A novel parameterization for feedback combination of contracting RNNs that enables direct optimization using standard deep learning libraries.
Revised: • A novel parameterization for feedback combination of contracting RNNs that enables directoptimization using standard deep learning libraries.

Original: • Novel contraction conditions for continuous-time nonlinear RNNs, to use in conjunction with the combination condition.
Revised: • Novel contraction conditions for continuous-time nonlinear RNNs, to use in conjunction withthe combination condition.

Original: We also identify ﬂaws in stability proofs from prior literature.
Revised: We also identify ﬂaws in stability proofs from prior literature.

Original: • Experiments demonstrating that our ‘network of networks’ sets a new state of the art for
Revised: • Experiments demonstrating that our ‘network of networks’ sets a new state of the art for

Original: stability-constrained RNNs on benchmark sequential processing tasks.
Revised: stability-constrained RNNs on benchmark sequential processing tasks

Original:  Network of Networks Model
Revised:  Network of Networks Model

Original: In this paper we analyze rate-based neural networks.
Revised: In this paper we analyze rate-based neural networks.

Original: Unlike spiking neural networks, these modelsare continuous and smooth.
Revised: Unlike spiking neural networks, these modelsare continuous and smooth.

Original: We consider the following RNN introduced by Wilson and Cowan [1972],which may be viewed as an approximation to a more biophysically-detailed spiking network:
Revised: We consider the following RNN introduced by Wilson and Cowan [1972],which may be viewed as an approximation to a more biophysically-detailed spiking network:

Original: [Equation]
Revised: [Equation]

Original: Here τ > 0 is the time-constant of the network [Dayan and Abbott, 2005].
Revised: Here τ > 0 is the time-constant of the network, typically between 10-100 milliseconds [Dayanand Abbott, 2005].

Original: The vector x ∈ R ncontains the voltages of all n neurons in the network.
Revised: The vector x ∈ R n contains the voltages of all n neurons in the network.

Original: The voltages are converted into ﬁring-ratesthrough a static nonlinearity φ .
Revised: Thevoltages are converted into ﬁring-rates through a static nonlinearity φ .

Original: We only consider monotonic activation functions with bounded slope:in other words, 0 ≤ φ (cid:48) ≤ g .
Revised: We only consider monotonicactivation functions with bounded slope: in other words, 0 ≤ φ (cid:48) ≤ g .

Original: We do not restrain the sign of the nonlinearity.
Revised: We do not restrain the sign ofthe nonlinearity.

Original: Common example nonlinearities φ ( x ) that satisfy these constraints are hyperbolic tangent and softplus.
Revised: Common example nonlinearities φ ( x ) that satisfy these constraints are hyperbolic tangent and softplus.

Original: The matrix W ∈ R n × n contains the synaptic weights of the RNN.
Revised: The matrix W ∈ R n × n contains the synaptic weights of the RNN.

Original: It is this matrix that ultimately determines thestability of (1), and will be a main target for our analysis.
Revised: It is thismatrix that ultimately determines the stability of (1), and will be a main target for our analysis.

Original: Finally, u ( t ) is the potentially time-varyingexternal input into the network, capturing both explicit input into the RNN from the external world,as well as unmodeled dynamics from other brain areas.
Revised: Finally, u ( t ) is the potentially time-varying external input into the network, capturing both explicitinput into the RNN from the external world, as well as unmodeled dynamics from other brain areas.

Original: Note that (1) is equivalent to another commonly used class of RNNs where the term Wx + uappears inside the nonlinearity.
Revised: Note that (1) is equivalent to another commonly used class of RNNs where the term Wx + uappears inside the nonlinearity.

Original: See Section A1.1 or [Miller and Fumarola, 2012] for details.
Revised: See Section A1.1 or [Miller and Fumarola, 2012] for details.

Original: Ourmathematical results apply equally well to both types of RNNs.
Revised: Ourmathematical results apply equally well to both types of RNNs.

Original: In order to extend (1) into a model of multiple interacting neural populations, we introduce the indexi , which runs from 1 to p , where p is the total number of RNNs in the collection of RNNs.
Revised: In order to extend (1) into a model of multiple interacting neural populations, we introduce the indexi , which runs from 1 to p , where p is the total number of RNNs in the collection of RNNs.

Original: For nowwe will assume linear interactions between RNNs, because this is the simplest case.
Revised: For nowwe will assume linear interactions between RNNs, because this is the simplest case.

Original: The linearityassumption can also be motivated by the fact that RNNs have been found to be well-approximated bylinear systems in many neuroscience contexts [Sussillo and Barak, 2013, Langdon and Engel, 2022].
Revised: Later on we willdiscuss extensions to nonlinear couplings.

Original: This  leads to the following equation for the ‘network of networks’:
Revised: This linearity assumptions leads to the following equationfor the ‘network of networks’:

Original: [Equation]
Revised: [Equation]

Original: where the matrix L ij captures the interaction between RNN i and RNN j .
Revised: where the matrix L ij captures the interaction between RNN i and RNN j .

Original: If RNN i has N i neuronsand RNN j has N j neurons, then L ij is an N i × N j matrix.
Revised: Since we allow the RNNsto be of different sizes, L ij is in general a rectangular matrix.

Original: We can now formalize the question posedin the abstract.
Revised: With this notation in hand, we cannow formalize the question posed in the abstract.

Original: Namely, "what types of connections between stable RNNs automatically preservestability?" becomes "what restrictions on L ij must be met in order to ensure overall stability of thenetwork of networks?".
Revised: Namely, "what types of connections betweenstable RNNs automatically preserve stability?" becomes "what restrictions on L ij must be met inorder to ensure overall stability of the network of networks?".

Original: We will now derive two combination ‘primitives’, negative feedback andhierarchical, which allow for recursive connection of contracting modules.
Revised: We will now derive two combination‘primitives’, negative feedback and hierarchical, which allow for recursive connection of contractingmodules.

Original: 2.1 Generalized Negative Feedback Between RNNs Preserves Stability
Revised: 2.1 Generalized Negative Feedback Between RNNs Preserves Stability

Original: We set aside for a moment the problem of determining when (1) is contracting.
Revised: We set aside for a moment the problem of determining when (1) is contracting.

Original: For now, assume thatwe have a collection of p contracting RNNs interacting through equation (2).
Revised: For now, assume thatwe have a collection of p contracting RNNs interacting through equation (2).

Original: Recall that contractionis deﬁned with respect to a metric , a way of measuring distances between trajectories in state space.
Revised: Recall that contractionis deﬁned with respect to a metric , a way of measuring distances between trajectories in state space.

Original: Thus, the i th RNN is contracting with respect to some metric M i .
Revised: Thus, the i th RNN is contracting with respect to some metric M i .

Original: We assume for simplicity that thismetric is constant.
Revised: We assume for simplicity that thismetric is constant.

Original: In the case where every RNN receives feedback from every other, we can preservestability by ensuring these connections are negative feedback.
Revised: In the case where every RNN receives feedback from every other, we can preservestability by ensuring these connections are negative feedback.

Original: In the simplest case where all RNNmodules are contracting in the identity, the negative feedback may be written as:
Revised: In the simplest case where all RNNmodules are contracting in the identity, the negative feedback may be written as:

Original: [Equation]
Revised: [Equation]

Original: This is a well known result from the contraction analysis literature [Slotine, 2003].
Revised: This is a well known result from the contraction analysis literature [Slotine, 2003].

Original: Our ﬁrst novelcontribution is a generalization and parameterization of this negative feedback which allows for directoptimization using gradient-based techniques.
Revised: Our ﬁrst novelcontribution is a generalization and parameterization of this negative feedback which allows for directoptimization using gradient-based techniques.

Original: In particular, if each L ij is parameterized as follows:
Revised: In particular, if each L ij is parameterized as follows:

Original: [Equation]
Revised: [Equation]

Original: for arbitrary matrix B ij , then the overall network of networks retains the assumed stability propertiesof the RNN subnetworks.
Revised: for arbitrary matrix B ij , then the overall network of networks retains the assumed stability propertiesof the RNN subnetworks.

Original: We provide a detailed proof in Section A3.1, but the basic idea relies onensuring that L ij = − L Tji in the appropriate metric .
Revised: We provide a detailed proof in Section A3.1, but the basic idea relies onensuring that L ij = − L Tji in the appropriate metric .

Original: This can be achieved via the constraint:
Revised: This can be achieved via the constraint:

Original: [Equation]
Revised: [Equation]

Original: Plugging (3) into the above expression veriﬁes that it is indeed satisﬁed.
Revised: Plugging (3) into the above expression veriﬁes that it is indeed satisﬁed.

Original: Because contraction analysisrelies on analyzing the symmetric part of Jacobian matrices, the skew-symmetry of L ij ‘cancels out’when computing the symmetric part, and leaves the stability of the subnetwork RNNs untouched.
Revised: Because contraction analysisrelies on analyzing the symmetric part of Jacobian matrices, the skew-symmetry of L ij ‘cancels out’when computing the symmetric part, and leaves the stability of the subnetwork RNNs untouched.

Original: Recursive Properties of Contracting Combinations
Revised: Recursive Properties of Contracting Combinations

Original: The feedback combination (3), taken together with hierarchical combinations, may be used as combination primitives for recursively constructing complicated network of networks while automatically maintaining stability.
Revised: The feedback combination (3), taken together with hierarchical combinations, may be used as combination primitives for recursively constructing complicated network of networks while automatically maintaining stability.

Original: The recursioncomes from the fact that once a modular system is shown to be contracting it may be treated as asingle contracting system, which may in turn be combined with other contracting systems, ad inﬁnitum .
Revised: The recursioncomes from the fact that once a modular system is shown to be contracting it may be treated as asingle contracting system, which may in turn be combined with other contracting systems, ad inﬁnitum .

Original: Note that while the feedback (3) requires linear interareal connections, hierarchical interarealconnections may be nonlinear[Lohmiller and Slotine, 1998].
Revised: Note that while the feedback (3) requires linear interareal connections, hierarchical interarealconnections may be nonlinear

Original:  Many Different Ways to Achieve Local Contraction
Revised:  Many Different Ways to Achieve Local Contraction

Original: In this section we return to the question of achieving contraction in the subnetwork RNNs.
Revised: In the previous section, we put aside the question of achieving contraction in the subnetwork RNNs.

Revised: In this section we return to it.

Original: Recallthat we wish to ﬁnd restrictions on W i such that the i th subnetwork RNN, described by (1), iscontracting.
Revised: Recall that we wish to ﬁnd restrictions on W i such that the i thsubnetwork RNN, described by (1), is contracting.

Original: Here we derive ﬁve such novel conditions (see Section A3 for detailed proofs).
Revised: Here we derive ﬁve such novel conditions (see Section A3 for detailed proofs).

Original: As wewill discuss, not all contraction conditions are equally useful - for example some conditions are easierto optimize or have higher model capacity than others.
Revised: As we will discuss, not all contraction conditions are equally usefulfor example some conditions are easier to optimize or have higher model capacity than others.

Original: In this section we also point out some ﬂawsin existing stability proofs in the RNN literature, and suggest some pathways towards correcting them.
Revised: In this section we also point out some ﬂaws in existing stability proofs in the RNN literature, andsuggest some pathways towards correcting them.

Original: Theorem 1 (Absolute Value Restricted Weights) .
Revised: Theorem 1 (Absolute Value Restricted Weights) .

Original: Let | W | denote the matrix formed by taking theelement-wise absolute value of W .
Revised: Let | W | denote the matrix formed by taking theelement-wise absolute value of W .

Original: If there exists a positive, diagonal P such that:
Revised: If there exists a positive, diagonal P such that:

Original: [Equation]
Revised: [Equation]

Original: then (1) is contracting in metric P .
Revised: then (1) is contracting in metric P .

Original: If W ii ≤ 0 , then | W | ii may be set to zero to reduce conservatism.
Revised: Moreover, if W ii ≤ 0 , then | W | ii may be set to zero to reduceconservatism.

Original: It is easy to ﬁnd matrices that satisfy Theorem 1, and given a matrix the condition is as easy tocheck as linear stability is.
Revised: It is easy to ﬁnd matrices that satisfy Theorem 1, and given a matrix the condition is as easy tocheck as linear stability is.

Original: Moreover, the condition guarantees we can obtain a metric that thesystem is known to contract in (see Section 4.1 for details).
Revised: Moreover, the condition guarantees we can obtain a metric that thesystem is known to contract in (see Section 4.1 for details).

Original: It is less straightforward to enforce thiscondition during training, however we found that subnetworks constrained by Theorem 1 can achievehigh performance in practice by simply ﬁxing W and only optimizing the connections betweensubnetworks (Section 4.2).
Revised: It is less straightforward to enforce thiscondition during training, however we found that subnetworks constrained by Theorem 1 can achievehigh performance in practice by simply ﬁxing W and only optimizing the connections betweensubnetworks (Section 4.2).

Original: As there are fewer parameters to optimize, this training technique is faster.
Revised: As there are fewer parameters to optimize, this training technique is faster.

Original: Theorem 2 (Symmetric Weights) .
Revised: Theorem 2 (Symmetric Weights) .

Original: If W = W T and g W ≺ I , then (1) is contracting.
Revised: If W = W T and g W ≺ I , then (1) is contracting.

Original: It has been known since the early 1990s that if (1) is autonomous (i.e the input u is constant) andhas symmetric weights with eigenvalues less than 1 /g , then there exists a unique ﬁxed point thatthe network converges to from any initial condition [Matsuoka, 1992].
Revised: It has been known since the early 1990s that if (1) is autonomous (i.e the input u is constant) andhas symmetric weights with eigenvalues less than 1 /g , then there exists a unique ﬁxed point thatthe network converges to from any initial condition [Matsuoka, 1992].

Original: Theorem 2 generalizes thisstatement to say that if (1) has symmetric weights with eigenvalues less than 1 /g , it is contracting.
Revised: Theorem 2 generalizes thisstatement to say that if (1) has symmetric weights with eigenvalues less than 1 /g , it is contracting.

Original: This includes previous results as a special case, because an autonomous contracting system has aunique ﬁxed point that the network converges to from any initial condition.
Revised: This includes previous results as a special case, because an autonomous contracting system has aunique ﬁxed point that the network converges to from any initial condition.

Original: Theorem 3 (Product of Diagonal and Orthogonal Weights) .
Revised: Theorem 3 (Product of Diagonal and Orthogonal Weights) .

Original: If there exists positive diagonal matrices
Revised: If there exists positive diagonal matrices

Original: P 1 and P 2 , as well as Q = Q T (cid:31) 

Revised: P 1 and P 2 , as well as Q = Q T (cid:31)

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: then (1) is contracting in metric M =
Revised: then (1) is contracting in metric M =

Original: ( P 1 QP 1 ) − 1 .
Revised: ( P 1 QP 1 ) − 1 .

Original: In contrast to the ﬁrst two contraction conditions, Theorem 3 is very easy to optimize.
Revised: In contrast to the ﬁrst two contraction conditions, Theorem 3 is very easy to optimize.

Original: To meetthe constraint that the P matrices are positive, one can parameterize their diagonal elements as
Revised: To meetthe constraint that the P matrices are positive, one can parameterize their diagonal elements as

Original: P ii =
Revised: P ii =

Original: d 2 i + (cid:15) , for some small positive constant (cid:15) , and optimize d i directly.
Revised: d 2 i + (cid:15) , for some small positive constant (cid:15) , and optimize d i directly.

Original: To meet the positivedeﬁniteness constraint on Q , one may parameterize it as Q = E T E
Revised: To meet the positivedeﬁniteness constraint on Q , one may parameterize it as Q = E T E

Original: + (cid:15) I and optimize E directly.
Revised: + (cid:15) I and optimize E directly.

Original: Theorem 4 (Triangular Weights) .
Revised: Theorem 4 (Triangular Weights) .

Original: If g W − I is triangular and Hurwitz, then (1) is contracting in adiagonal metric.
Revised: If g W − I is triangular and Hurwitz, then (1) is contracting in adiagonal metric.

Original: Theorem 4 follows from the fact that a hierarchy of contracting systems is also contracting.
Revised: Theorem 4 follows from the fact that a hierarchy of contracting systems is also contracting.

Original: Theorem 5 (Singular Value Restricted Weights) .
Revised: Theorem 5 (Singular Value Restricted Weights) .

Original: If there exists a positive diagonal matrix P suchthat:
Revised: If there exists a positive diagonal matrix P suchthat:

Original: [Equation]
Revised: [Equation]

Original: then (1) is contracting in metric P .
Revised: then (1) is contracting in metric P .

Original: In the case of discrete-time RNNs, this contraction condition has been proved by many differentauthors in many different settings.
Revised: In the case of discrete-time RNNs, this contraction condition has been proved by many differentauthors in many different settings.

Original: When P =
Revised: When P =

Original: I , it is known as the echo-state condition for discretetime RNNs [Jaeger, 2001].
Revised: I , it is known as the echo-state condition for discretetime RNNs [Jaeger, 2001].

Original: This was then generalized to diagonal P in Buehner and Young [2006].
Revised: This was then generalized to diagonal P in Buehner and Young [2006].

Original: More recently, the original echo-state condition was rediscovered by Miller and Hardt [2018] in themachine learning literature.
Revised: More recently, the original echo-state condition was rediscovered by Miller and Hardt [2018] in themachine learning literature.

Original: Following this rediscovery, the condition was generalized to P (cid:54)
Revised: Following this rediscovery, the condition was generalized to P (cid:54)

Original: =
Revised: =

Original: I in
Revised: I in

Original: Revay and Manchester [2020].
Revised: Revay and Manchester [2020].

Original: Here we show that it applies to continuous-time RNNs as well.
Revised: Here we show that it applies to continuous-time RNNs as well.

Original: Inaddition to sharing a contraction condition, echo-state networks are related to our Sparse Combination

Original: Networks through the sparsity condition.

Original: However, in the echo-state approach only the outgoingweights are optimized, while in our approach we train the internal weights of the network.

Original: What do the Jacobian Eigenvalues Tell Us?
Revised: What do the Jacobian Eigenvalues Tell Us?

Original: Several recent papers in ML, e.g [Haber and Ruthotto,2017, Chang et al., 2019], claim that a sufﬁcient condition for stability of the nonlinear system:˙ x = f ( x , t ) is that the associated Jacobian matrix J ( x , t ) =
Revised: Several recent papers in machine learning, e.g [Haberand Ruthotto, 2017, Chang et al., 2019], claim that a sufﬁcient condition for stability of the nonlinearsystem: [Equation] is that the associated Jacobian matrix J ( x , t ) =

Original: ∂ f ∂ x has eigenvalues whose real parts are strictlynegative, i.e:
Revised: ∂ f ∂ x has eigenvalues whose real parts are strictlynegative, i.e:

Original: [Equation]
Revised: [Equation]

Original: with α > 0 .
Revised: with α > 0 .

Original: Unfortunately, this claim is generally false (see Section 4.4.2 in [Slotine and Li, 1991]).
Revised: Unfortunately, this claim is generally false. For a counter-example, see Section 4.4.2 in[Slotine and Li, 1991].

Original: However, in the speciﬁc case of the RNN (1), it appears that the eigenvalues of the symmetric part of
Revised: However, in the speciﬁc case of the RNN (1), it appears that the eigenvalues of the symmetric part of

Original: W do provide information on global stability in a number of applications.
Revised: W do provide information on global stability in a number of applications.

Original: For example, in [Matsuoka,1992] it was shown that if W s = 12 ( W + W T ) has all its eigenvalues less than unity, and u isconstant, then (1) has a unique, globally asymptotically stable ﬁxed point.
Revised: For example, in [Matsuoka,1992] it was shown that if W s = 12 ( W + W T ) has all its eigenvalues less than unity, and u isconstant, then (1) has a unique, globally asymptotically stable ﬁxed point.

Original: This condition also impliesthat the real parts of the eigenvalues of the Jacobian are uniformly negative.
Revised: This condition also impliesthat the real parts of the eigenvalues of the Jacobian are uniformly negative.

Original: Moreover, in [Changet al., 2019] it was shown that setting the symmetric part of W s = 12 ( W + W T ) almost equal to zero(yet slightly negative) led to rotational, yet stable dynamics in practice.
Revised: Moreover, in [Changet al., 2019] it was shown that setting the symmetric part of W s = 12 ( W + W T ) almost equal to zero(yet slightly negative) led to rotational, yet stable dynamics in practice.

Original: This leads us to the followingtheorem, which shows that if the slopes of the activation functions change sufﬁciently slowly as afunction of time, then the condition in [Matsuoka, 1992] in fact implies global contraction of (1).
Revised: This leads us to the followingtheorem, which shows that if the slopes of the activation functions change sufﬁciently slowly as afunction of time, then the condition in [Matsuoka, 1992] in fact implies global contraction of (1).

Original: Theorem 6.
Revised: Theorem 6.

Original: Let D be a positive, diagonal matrix with D ii
Revised: Let D be a positive, diagonal matrix with D ii

Original: = dφ i dx i , and let P be an arbitrary, positivediagonal matrix.
Revised: = dφ i dx i , and let P be an arbitrary, positivediagonal matrix.

Original: If:
Revised: If:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: for c, β > 0 , then (1) is contracting in metric D with rate β .
Revised: for c, β > 0 , then (1) is contracting in metric D with rate β .

Original: We stress however, that it is an open question whether or not diagonal stability of W implies that(1) is contracting.
Revised: We stress however, that it is an open question whether or not diagonal stability of W implies that(1) is contracting.

Original: It has been conjectured that diagonal stability of g W − I is a sufﬁcient conditionfor global contraction of (1)
Revised: It has been conjectured that diagonal stability of g W − I is a sufﬁcient conditionfor global contraction of (1)

Original: [Revay et al., 2020], however this has been difﬁcult to prove.
Revised: [Revay et al., 2020], however this has been difﬁcult to prove.

Original: To bettercharacterize this conjecture, we present Theorem 7, which shows by way of counterexample thatdiagonal stability of g W − I does not imply global contraction in a constant metric for (1).
Revised: To bettercharacterize this conjecture, we present Theorem 7, which shows by way of counterexample thatdiagonal stability of g W − I does not imply global contraction in a constant metric for (1).

Original: Theorem 7.
Revised: Theorem 7.

Original: Satisfaction of the condition g W sym − I ≺is not sufﬁcient to show globalcontraction of the general nonlinear RNN (1) in any constant metric.
Revised: Satisfaction of the condition g W sym − I ≺is not sufﬁcient to show globalcontraction of the general nonlinear RNN (1) in any constant metric.

Original: High levels of antisymmetry in
Revised: High levels of antisymmetry in

Original: W can make it impossible to ﬁnd such a metric, which we demonstrate via a 2 × 2 counterexample ofthe following form, with c ≥ 2 :
Revised: W can make it impossible to ﬁnd such a metric, which we demonstrate via a 2 × 2 counterexample ofthe following form, with c ≥ 2 :

Original: [Equation]
Revised: [Equation]

Original:  Stability-Constrained Network of Networks Perform Well on Benchmarks
Revised:  Stability-Constrained Network of Networks Perform Well on Benchmarks

Original: A natural concern is that stability of an RNN may come at the cost of its expressivity, which isparticularly relevant for integrating information over long timescales.
Revised: A natural concern is that stability of an RNN may come at the cost of its expressivity, which isparticularly relevant for integrating information over long timescales.

Original: To investigate whether thismight be an issue for our model, we trained a stability-constrained network-of-networks on threebenchmark sequential image classiﬁcation tasks: sequential MNIST, permuted seqMNIST, andsequential CIFAR10.
Revised: To investigate whether thismight be an issue for our model, we trained a stability-constrained network-of-networks on threebenchmark sequential image classiﬁcation tasks: sequential MNIST, permuted seqMNIST, andsequential CIFAR10.

Original: These tasks are often used to measure information processing ability over longsequences [Le et al., 2015].
Revised: These tasks are often used to measure information processing ability over longsequences [Le et al., 2015].

Original: Images are presented pixel-by-pixel, and the network makes a predictionat the end of the sequence.
Revised: Images are presented pixel-by-pixel, and the network makes a predictionat the end of the sequence.

Original: In permuted seqMNIST, pixels are input in a ﬁxed but random order.
Revised: In permuted seqMNIST, pixels are input in a ﬁxed but random order.

Original: All of our experiments were done on networks governed by (2).
Revised: All of our experiments were done on networks governed by (2).

Original: The nonlinear subnetwork RNNswere connected to each other via linear all-to-all negative feedback, given by (3).
Revised: The nonlinear subnetwork RNNswere connected to each other via linear all-to-all negative feedback, given by (3).

Original: For all subnetworkswe use the ReLU activation function.
Revised: For all subnetworkswe use the ReLU activation function.

Original: To enforce contraction of each individual subnetwork, wefocused on two stability constraints from our theoretical results: Theorems 1 and 5.
Revised: To enforce contraction of each individual subnetwork, wefocused on two stability constraints from our theoretical results: Theorems 1 and 5.

Original: In the case of
Revised: In the case of

Original: Thm 1, we did not train the individual subnetworks’ weight matrices, but only trained the connectionsbetween subnetworks (Figure 2B).
Revised: Thm 1, we did not train the individual subnetworks’ weight matrices, but only trained the connectionsbetween subnetworks (Figure 2A).

Original: For Thm 5, we trained all parameters of the model (Figure 2C).
Revised: For Thm 5, we trained all parameters of the model (Figure 2B).

Original: We refer to networks with subnetworks constrained by Theorem 1 as ‘Sparse Combo Nets’ andto networks with subnetworks constrained by Theorem 5 as ‘SVD Combo Nets’.
Revised: We refer to networks with subnetworks constrained by Theorem 1 as ‘Sparse Combo Nets’ andto networks with subnetworks constrained by Theorem 5 as ‘SVD Combo Nets’.

Original: Throughout theexperimental results we use the notation ‘ p × n network’ - such a network consists of p distinctsubnetwork RNNs, with each such subnetwork RNN containing n units.
Revised: Throughout the experimental results we use the notation ‘ p × n network’ - such a network consists of p distinctsubnetwork RNNs, with each such subnetwork RNN containing n units.

Original: [Figure]
Revised: [Figure]

Revised: constrained

Revised: Theorem sparse initialization

Revised: Linear negative feedback connections were trained between the subnetworks according to (3).

Original: 4.1 Network Initialization and Training
Revised: 4.1 Network Initialization and Training

Original: [Figure] For the Sparse Combo Net we were not able to ﬁnd a parameterization to use during training, but it iseasy to randomly generate matrices with a particular likelihood of meeting the Theorem 1 condition,by selecting an appropriate sparsity level and limit on entry magnitude.
Revised: For the Sparse Combo Net we were not able to ﬁnd a parameterization to use during training, but it iseasy to randomly generate matrices with a particular likelihood of meeting the Theorem 1 condition,by selecting an appropriate sparsity level and limit on entry magnitude.

Original: Sparsity in particular isof interest due to its relevance in neurobiology and machine learning, so it is convenient that thecondition makes it easy to verify stability of many different sparse RNNs.
Revised: Sparsity in particular isof interest due to its relevance in neurobiology and machine learning, so it is convenient that thecondition makes it easy to verify stability of many different sparse RNNs.

Original: As g = 1 for ReLUactivation, we check potential subnetwork matrices W by simply verifying linear stability of | W |− I .
Revised: As g = 1 for ReLUactivation, we check potential subnetwork matrices W by simply verifying linear stability of | W |− I .

Original: Because every RNN meeting the condition has a corresponding well-deﬁned stable LTI systemcontracting in the same metric, it is easy to ﬁnd a metric to use in our training algorithm: solvingfor M in − I =
Revised: Because every RNN meeting the condition has a corresponding well-deﬁned stable LTI systemcontracting in the same metric, it is easy to ﬁnd a metric to use in our training algorithm: solvingfor M in − I =

Original: MA + A T M will produce a valid metric for any stable LTI system
Revised: MA + A T M will produce a valid metric for any stable LTI system

Original: A [Slotine and
Revised: A [Slotine and

Original: Li, 1991].
Revised: Li, 1991].

Original: We also utilize the fact that Hurwitz Metzler matrices are diagonally stable to improveefﬁciency of computing M (as well as in our proof of Theorem 1).
Revised: We also utilize the fact that Hurwitz Metzler matrices are diagonally stable to improveefﬁciency of computing M (as well as in our proof of Theorem 1).

Original: See Section A4.1.1 for details.
Revised: See Section A4.1.1 for details.

Original: We therefore randomly generated ﬁxed subnetworks satisfying Theorem 1 and trained only the linearconnections between them (Figure 3), as well as the linear input and output layers.
Revised: We therefore randomly generated ﬁxed subnetworks satisfying Theorem 1 and trained only the linearconnections between them (Figure 3), as well as the linear input and output layers.

Original: More informationon network initialization, hyperparameter tuning, and training algorithm is provided in Section A4.
Revised: More informationon network initialization, hyperparameter tuning, and training algorithm is provided in Section A4.

Original: [Figure]
Revised: [Figure]

Original: For the SVD Combo Net on the other hand, we ensured contraction of each subnetwork  bydirect parameterization (described in Section A5), thus allowing all weights to be trained.
Revised: For the SVD Combo Net on the other hand, we ensured contraction of each subnetwork RNN bydirect parameterization (described in Section A5), thus allowing all weights to be trained.

Original: 4.2 Results
Revised: 4.2 Results

Original: The Sparse Combo Net architecture achieved the highest overall performance on both permutedseqMNIST and seqCIFAR10, with 96.94% and 65.72% best test accuracies respectively - therebysetting a new SOTA for stable RNNs (Table 1).
Revised: The Sparse Combo Net architecture achieved the highest overall performance on both permutedseqMNIST and seqCIFAR10, with 96.94% and 65.72% best test accuracies respectively - therebysetting a new SOTA for stable RNNs (Table 1).

Original: Furthermore, we were able to reproduce SOTAscores over several repetitions, including 10 trials of seqCIFAR10.
Revised: Furthermore, we were able to reproduce SOTAscores over several repetitions, including 10 trials of seqCIFAR10.

Original: Along with repeatability of results,we also show that the contraction constraint on the connections between subnetworks ( L in (3)) isimportant for performance, particularly in the Sparse Combo Net (Section 4.2.3).
Revised: Along with repeatability of results,we also show that the contraction constraint on the connections between subnetworks ( L in (3)) isimportant for performance, particularly in the Sparse Combo Net (Section 4.2.3).

Original: Additionally, we proﬁle how various architecture settings impact performance of our networks.
Revised: Additionally, we proﬁle how various architecture settings impact performance of our networks.

Original: Inboth networks, we found that increasing the total number of neurons improved task performance, butwith diminishing returns (Section 4.2.1).
Revised: Inboth networks, we found that increasing the total number of neurons improved task performance, butwith diminishing returns (Section 4.2.1).

Original: We also found that the sparsity of the hidden-to-hiddenweights in Sparse Combo Net had a large impact on the ﬁnal network performance (Section 4.2.2) [Table]
Revised: We also found that the sparsity of the hidden-to-hiddenweights in Sparse Combo Net had a large impact on the ﬁnal network performance (Section 4.2.2).

Original: [Table]
Revised: [Table]

Original: best test accuracy.

Original: Architectures are grouped into three categories: baselines, best performing RNNs with claimed stability guarantee, and networks achieving overall SOTA.

Original: Within each grouping, networks are ordered by number of trainable parameters (for CIFAR10 if it differed across tasks).

Original: Our network is highlighted.
Revised: Our network is highlighted.

Original: Where possible, we include information on repeatability.
Revised: Where possible, we include information on repeatability.

Original: 4.2.1 Experiments with Network Size
Revised: 4.2.1 Experiments with Network Size

Original: Understanding the effect of size on network performance is important to practical application ofthese architectures.
Revised: Understanding the effect of size on network performance is important to practical application ofthese architectures.

Original: For both Sparse Combo Net and SVD Combo Net, increasing the number ofsubnetworks while holding other settings constant (including ﬁxing the size of each subnetwork at 32units) was able to increase network test accuracy on permuted seqMNIST to a point (Figure 4).
Revised: For both Sparse Combo Net and SVD Combo Net, increasing the number ofsubnetworks while holding other settings constant (including ﬁxing the size of each subnetwork at 32units) was able to increase network test accuracy on permuted seqMNIST to a point (Figure 4).

Original: The greatest performance jump happened when increasing from one module (37.1% Sparse Combo
Revised: The greatest performance jump happened when increasing from one module (37.1% Sparse Combo

Original: Net, 61.8% SVD Combo Net) to two modules (89.1% Sparse Combo Net, 92.9% SVD Combo Net).
Revised: Net, 61.8% SVD Combo Net) to two modules (89.1% Sparse Combo Net, 92.9% SVD Combo Net).

Original: After that the performance increased steadily with number of modules until saturating at ∼ 97% for
Revised: After that the performance increased steadily with number of modules until saturating at ∼ 97% for

Original: Sparse Combo Net and ∼ 95% for SVD Combo Net.
Revised: Sparse Combo Net and ∼ 95% for SVD Combo Net.

Original: As the internal subnetwork weights are not trained in Sparse Combo Net, it is unsurprising that itsperformance was substantially worse at the smallest sizes.
Revised: As the internal subnetwork weights are not trained in Sparse Combo Net, it is unsurprising that itsperformance was substantially worse at the smallest sizes.

Original: However Sparse Combo Net surpasses
Revised: However Sparse Combo Net surpasses

Original: SVD Combo Net by the 12 × 32 network size, which contains a modest 384 total units.
Revised: SVD Combo Net by the 12 × 32 network size, which contains a modest 384 total units.

Original: Due to thebetter performance of the Sparse Combo Net, we focused additional analyses there.
Revised: Due to thebetter performance of the Sparse Combo Net, we focused additional analyses there.

Original: Note also thatthe SVD Combo Net never reached 55% test accuracy for CIFAR10 in our early experiments, whilethe Sparse Combo Net easily surpassed that.
Revised: For more info on SVD Combo Net results, see Section A5.

Original: 4.2.2 Experiments with Sparsity
Revised: 4.2.2 Experiments with Sparsity

Original: Because of the link between sparsity and stability as well as the biological relevance of sparsity, weexplored in detail how subnetwork sparsity affects the performance of Sparse Combo Net.
Revised: Because of the link between sparsity and stability as well as the biological relevance of sparsity, weexplored in detail how subnetwork sparsity affects the performance of Sparse Combo Net.

Original: We ran anumber of experiments on the permuted seqMNIST task, varying sparsity level while holding networksize and other hyperparameters constant.
Revised: We ran anumber of experiments on the permuted seqMNIST task, varying sparsity level while holding networksize and other hyperparameters constant.

Original: Here we use " n % sparsity level" to refer to a network withsubnetworks that have just n % of their weights non-zero.
Revised: Here we use " n % sparsity level" to refer to a network withsubnetworks that have just n % of their weights non-zero.

Original: We observed a large ( > 5 percentage point) performance boost when switching from a 26 .
Revised: We observed a large ( > 5 percentage point) performance boost when switching from a 26 .

Original: 5% sparsitylevel to a 10% sparsity level in the 11 × 32 Sparse Combo Net (Figure 5), and subsequently decidedto test signiﬁcantly sparser subnetworks in a 16 × 32 Sparse Combo Net.
Revised: 5% sparsitylevel to a 10% sparsity level in the 11 × 32 Sparse Combo Net (Figure 5), and subsequently decidedto test signiﬁcantly sparser subnetworks in a 16 × 32 Sparse Combo Net.

Original: We trained networkswith sparsity levels of 5% , 3 . 3% , and 1% , as well as 10% for baseline comparison (Figure S2A).
Revised: We trained networkswith sparsity levels of 5% , 3 . 3% , and 1% , as well as 10% for baseline comparison (Figure S2A).

Original: A3 .
Revised: A3 .

Original: 3% sparsity level produced the best results, leading to our SOTA performance on both permutedseqMNIST and seqCIFAR10.
Revised: 3% sparsity level produced the best results, leading to our SOTA performance on both permutedseqMNIST and seqCIFAR10.

Original: For a component RNN size of just 32 units, this sparsity level is quitesmall, with only one or two directional connections per neuron on average (Figure S7).
Revised: For a component RNN size of just 32 units, this sparsity level is quitesmall, with only one or two directional connections per neuron on average (Figure S7).

Revised: performance over the course of training for two 11 × 32 Sparse

Original:  Nets with different sparsity levels.
Revised: Combo Nets with different sparsity levels.

Original: The use of sparsity in subnetworks to improve performance suggests another interesting directionthat could enable better scalability of total network size - enforcing sparsity in the linear feedbackweight matrix ( L ).
Revised: The use of sparsity in subnetworks to improve performance suggests another interesting directionthat could enable better scalability of total network size - enforcing sparsity in the linear feedbackweight matrix ( L ).

Original: We performed pilot testing of this idea, which showed promise in mitigating thesaturation effect seen in Figure 4.
Revised: We performed pilot testing of this idea, which showed promise in mitigating thesaturation effect seen in Figure 4.

Original: Those results are detailed in Section A4.2.

Revised: Those results are detailed in Section A4.2.

Original: (Table S1).
Revised: (Table S1).

Original: We alsodiscuss related future directions in Section A2.2.
Revised: We alsodiscuss related future directions in Section A2.2.

Original: 4.2.3 Repeatability and Controls
Revised: 4.2.3 Repeatability and Controls

Original: Sparse Combo Net does not have the connections within its subnetworks trained, so network performance could be particularly susceptible to random initialization.
Revised: Sparse Combo Net does not have the connections within its subnetworks trained, so network performance could be particularly susceptible to random initialization.

Original: Thus we ran repeatability studies onpermuted sequential MNIST and sequential CIFAR10 using our best network settings ( 16 × 32 with subnetwork sparsity level of 3 . 3% ) and an extended training period.
Revised: Thus we ran repeatability studies on permuted sequential MNIST and sequential CIFAR10 using our best network settings ( 16 × 32 withsubnetwork sparsity level of 3 . 3% ) and an extended training period.

Original: Mean performance over 4 trialsof permuted seqMNIST was 96.85% with 0.019 variance, while mean performance over 10 trials ofseqCIFAR10 was 64.72% with 0.406 variance.
Revised: Mean performance over 4 trialsof permuted seqMNIST was 96.85% with 0.019 variance, while mean performance over 10 trials ofseqCIFAR10 was 64.72% with 0.406 variance.

Original: Note we also ran a number of additional experimentson size and sparsity settings, described in Section A4.2.
Revised: Note we also ran a number of additional experimentson size and sparsity settings, described in Section A4.2.

Original: Across the permuted seqMNIST trials, best test accuracy always fell between 96 .
Revised: Across the permuted seqMNIST trials, best test accuracy always fell between 96 .

Original: 65% and 96 .
Revised: 65% and 96 .

Original: 94% ,a range much smaller than the differences seen with changing sparsity settings and network size.
Revised: 94% ,a range much smaller than the differences seen with changing sparsity settings and network size.

Original: Three of the four trials showed best test accuracy ≥ 96 .
Revised: Three of the four trials showed best test accuracy ≥ 96 .

Original: 88% , despite some variability in early trainingperformance (Figure S3).
Revised: 88% , despite some variability in early trainingperformance (Figure S3).

Original: Similarly, eight of the ten seqCIFAR10 trials had test accuracy > 64 .
Revised: Similarly, eight of the ten seqCIFAR10 trials had test accuracy > 64 .

Original: 3% ,with all results falling between 63.73% and 65.72% (Figure S4).
Revised: 3% ,with all results falling between 63.73% and 65.72% (Figure S4).

Original: This robustly establishes a new
Revised: This robustly establishes a new

Original: SOTA for stable RNNs, comfortably beating the previously reported (single run) 64.2% test accuracyachieved by Lipschitz RNN [Erichson et al., 2021].
Revised: SOTA for stable RNNs, comfortably beating the previously reported (single run) 64.2% test accuracyachieved by Lipschitz RNN [Erichson et al., 2021].

Original: As a control study, we also tested how sensitive the Sparse Combo Net was to the stabilizationcondition on the interconnection matrix ( L in (3)).
Revised: As a control study, we also tested how sensitive the Sparse Combo Net was to the stabilizationcondition on the interconnection matrix ( L in (3)).

Original: To do so, we initialized the individual RNNmodules in a 24 × 32 network as before, but set L = B and did not constrain B at all during training,thus no longer ensuring contraction of the overall system.
Revised: To do so, we initialized the individual RNNmodules in a 24 × 32 network as before, but set L = B and did not constrain B at all during training,thus no longer ensuring contraction of the overall system.

Original: This resulted in 47.0% test accuracyon the permuted seqMNIST task, a stark decrease from the original 96.7% test accuracy - therebydemonstrating the utility of the contraction condition.
Revised: This resulted in 47.0% test accuracyon the permuted seqMNIST task, a stark decrease from the original 96.7% test accuracy - therebydemonstrating the utility of the contraction condition.

Original:  Discussion
Revised:  Discussion

Original: The question of neural stability is one of the oldest questions in computational neuroscience.
Revised: The question of neural stability is one of the oldest questions in computational neuroscience.

Original: Indeed,cyberneticists were concerned with this question before the term ‘computational neuroscience’ existed[Wiener, 1948, Ashby, 1952].
Revised: Indeed,cyberneticists were concerned with this question before the term ‘computational neuroscience’ existed[Wiener, 1948, Ashby, 1952].

Original: Stability is a central component in several inﬂuential neurosciencetheories [Hopﬁeld, 1982, Seung, 1996, Murphy and Miller, 2009], perhaps the most well-knownbeing that memories are stored as stable point attractors [Hopﬁeld, 1982].
Revised: Stability is a central component in several inﬂuential neurosciencetheories [Hopﬁeld, 1982, Seung, 1996, Murphy and Miller, 2009], perhaps the most well-knownbeing that memories are stored as stable point attractors [Hopﬁeld, 1982].

Original: Engineers have long noted that stability and modularity are related [Khalil, 2002] - for example,passive dynamical systems can be connected in feedback combinations in ways that automaticallypreserve stability [Slotine and Li, 1991].
Revised: Engineers have long noted that stability and modularity are related [Khalil, 2002] - for example,passive dynamical systems can be connected in feedback combinations in ways that automaticallypreserve stability [Slotine and Li, 1991].

Original: Curiously, this formal connection between stability and‘combinability" was unexplored in the context of computational neuroscience.
Revised: Curiously, this formal connection between stability andmodularity was unexplored in the context of computational neuroscience.

Original: We begin to ﬁll thatgap by providing novel stability conditions for feedback systems consisting of many interacting RNNs.
Revised: We begin to ﬁll that gap byproviding novel stability conditions for feedback systems consisting of many interacting RNNs.

Original: While primarily motivated by neuroscience, our approach is also relevant for machine learning.
Revised: While primarily motivated by neuroscience, our approach is also relevant for machine learning.

Original: Deeplearning models are often as inscrutable as they are powerful.
Revised: Deeplearning models are often as inscrutable as they are powerful.

Original: This opaqueness can be dangerous insafety-critical applications, such as autonomous driving or human-centered robotics, and it limitsconceptual progress.
Revised: This opaqueness can be dangerous insafety-critical applications, such as autonomous driving or human-centered robotics, and it limitsconceptual progress.

Original: In the case of recurrent models, one difﬁculty is that providing a certiﬁcate ofstability is currently impossible or computationally impractical.
Revised: In the case of recurrent models, one difﬁculty is that providing a certiﬁcate ofstability is currently impossible or computationally impractical.

Original: Given that stability is a fundamentalproperty of dynamical systems – and is intimately linked to concepts of control, generalization,efﬁciency, and robustness – the ability to guarantee stability of a recurrent model will be importantfor ensuring deep models behave as we expect them to.
Revised: Given that stability is a fundamentalproperty of dynamical systems – and is intimately linked to concepts of control, generalization,efﬁciency, and robustness – the ability to guarantee stability of a recurrent model will be importantfor ensuring deep models behave as we expect them to.

Original: Moreover, advanced applications of modern machine learning will likely involve combinations oftrained networks, as are already used in systems such as DeepMind’s AlphaGo.
Revised: Moreover, advanced applications of modern machine learning will likely involve combinations oftrained networks, as are already used in systems such as DeepMind’s AlphaGo.

Original: Recursively buildingsuch combinations in an effective and stable fashion while also allowing for continual reﬁnement ofthe individual networks - as nature does for biological networks - will require new analysis tools.
Revised: Recursively buildingsuch combinations in an effective and stable fashion while also allowing for continual reﬁnement ofthe individual networks - as nature does for biological networks - will require new analysis tools.

Original: Herewe have taken a concrete step towards the development of such tools, not only through our theoreticalresults, but also through their application to create stable combination network architectures thatperform well in practice on benchmark tasks.
Revised: Herewe have taken a concrete step towards the development of such tools, not only through our theoreticalresults, but also through their application to create stable combination network architectures thatperform well in practice on benchmark tasks.

Original: To summarize, we have theoretically characterized conditions under which ‘multi-area RNNs’ (i.e
Revised: To summarize, we have theoretically characterized conditions under which ‘multi-area RNNs’ (i.e

Original: RNNs of RNNs) will be stable, given that each individual RNN is stable.
Revised: RNNs of RNNs) will be stable, given that each individual RNN is stable.

Original: We have also providedseveral novel stability conditions for individual RNNs which are compatible with these stabilitypreserving interareal connections.
Revised: We have also providedseveral novel stability conditions for individual RNNs which are compatible with these stabilitypreserving interareal connections.

Original: We parameterized our stability conditions for easy optimizationusing gradient-based techniques.
Revised: We parameterized our stability conditions for easy optimizationusing gradient-based techniques.

Original: We then leveraged this parameterization to successfully trainrecurrently connected ‘network of networks’ on three challenging sequential processing benchmarks,demonstrating that stability does not necessarily come at the cost of expressivity.
Revised: We then leveraged this parameterization to successfully trainrecurrently connected ‘network of networks’ on three challenging sequential processing benchmarks,demonstrating that stability does not necessarily come at the cost of expressivity.

Original: All together, ourresults contribute progress towards understanding how the brain (i.e a ‘RNN of RNNs’) maintainsstable and accurate function in the presence of massive interareal feedback, as well as external input.
Revised: All together, ourresults contribute progress towards understanding how the brain (i.e a ‘RNN of RNNs’) maintainsstable and accurate function in the presence of massive interareal feedback, as well as external input.

Original: Appendix
Revised: AppendixA1 Extended Background

Original: A1 Extended Background

Original: A1.1 Two Different RNNs . .
Revised: A1.1 Two Different RNNs . .

Original: A1.2 Contraction Math . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A1.2 Contraction Math . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A1.2.1 Feedback and Hierarchical Combinations . . . . . . . . . . . . . . . . . .
Revised: A1.2.1 Feedback and Hierarchical Combinations . . . . . . . . . . . . . . . . . .

Original: A2 Extended Discussion 

Revised: A2 Extended Discussion

Original: A2.1 Limitations . .
Revised: A2.1 Limitations . .

Original: A2.2 Future Directions . .
Revised: A2.2 Future Directions . .

Original: A
Revised: A

Original: Proofs for Main Results
Revised: Proofs for Main Results

Original: A3.1 Proof of Feedback Combination Property . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.1 Proof of Feedback Combination Property . . . . . . . . . . . . . . . . . . . . . .

Original: A3.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A3.3 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.3 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A3.4 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.4 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A3.5 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.5 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A3.6 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.6 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A3.7 Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A3.7 Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A3.8 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .A
Revised: A3.8 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original:  Sparse Combo Net Details {+
+}
Revised: A4 Sparse Combo Net Details {+
+}

Original: A4.1 Extended Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.1 Extended Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.1.1 Initialization and Training . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.1.1 Initialization and Training . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.1.2 Code and Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.1.2 Code and Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.2 Extended Results . . .
Revised: A4.2 Extended Results . . .

Original: A4.2.1 Network Size and Modularity Comparison . . . . . . . . . . . . . . . . .
Revised: A4.2.1 Network Size and Modularity Comparison . . . . . . . . . . . . . . . . .

Original: A4.2.2 Sparsity Settings Comparison . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.2.2 Sparsity Settings Comparison . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.2.3 Repeatability Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.2.3 Repeatability Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.2.4 Scalability Pilot: Feedback Sparsity Tests . . . . . . . . . . . . . . . . . .
Revised: A4.2.4 Scalability Pilot: Feedback Sparsity Tests . . . . . . . . . . . . . . . . . .

Original: A4.3 Architecture Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.3 Architecture Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.3.1 Number of Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.3.1 Number of Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.3.2 Inspecting Trained Network Weights . . . . . . . . . . . . . . . . . . . . .
Revised: A4.3.2 Inspecting Trained Network Weights . . . . . . . . . . . . . . . . . . . . .

Original: A4.4 Tables of Results by Trial (Supplementary Documentation) . . . . . . . . . . . . .
Revised: A4.4 Tables of Results by Trial (Supplementary Documentation) . . . . . . . . . . . . .

Original: A4.4.1 Permuted seqMNIST Trials . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A4.4.1 Permuted seqMNIST Trials . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A4.4.2 seqCIFAR10 Trials . .
Revised: A4.4.2 seqCIFAR10 Trials . .

Original: A5 SVD Combo Net Details 

Revised: A5 SVD Combo Net Details

Original: A5.1 Parameterization Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A5.1 Parameterization Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A5.2 Control Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A5.2 Control Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A5.3 Model Code . .
Revised: A5.3 Model Code . .

Original: A Extended BackgroundA1.1 Two Different RNNs
Revised: A Extended BackgroundA1.1 Two Different RNNs

Original: Note that in neuroscience, the variable x in equation (1) is typically thought of as a vector of neuralmembrane potentials.
Revised: Note that in neuroscience, the variable x in equation (1) is typically thought of as a vector of neuralmembrane potentials.

Original: It was shown in [Miller and Fumarola, 2012] that the RNN (1) is equivalent viaan afﬁne transformation to another commonly used RNN model,
Revised: It was shown in [Miller and Fumarola, 2012] that the RNN (1) is equivalent viaan afﬁne transformation to another commonly used RNN model,

Original: [Equation]
Revised: [Equation]

Original: where the variable y is interpreted as a vector of ﬁring rates, rather than membrane potentials.
Revised: where the variable y is interpreted as a vector of ﬁring rates, rather than membrane potentials.

Original: Thetwo models are related by the transformation x
Revised: Thetwo models are related by the transformation x

Original: =
Revised: =

Original: Wy + b , which yields
Revised: Wy + b , which yields

Original: [Equation]
Revised: [Equation]

Original: where v ≡ b + τ ˙ b .
Revised: where v ≡ b + τ ˙ b .

Original: Thus b is a low-pass ﬁltered version of v (or conversely, v may be viewedas a ﬁrst order prediction of b ) and the contraction properties of the system are unaffectedby the afﬁne transformation.
Revised: Thus b is a low-pass ﬁltered version of v (or conversely, v may be viewedas a ﬁrst order prediction of b ) and the contraction properties of the system are unaffectedby the afﬁne transformation.

Original: Note that the above equivalence holds even in the case where
Revised: Note that the above equivalence holds even in the case where

Original: W is not invertible.
Revised: W is not invertible.

Original: In this case, the two models are proven to be equivalent, provided thatb (0) and y (0) satisfy certain conditions–which are always possible to satisfy [Miller and
Revised: In this case, the two models are proven to be equivalent, provided thatb (0) and y (0) satisfy certain conditions–which are always possible to satisfy [Miller and

Original: Fumarola, 2012].
Revised: Fumarola, 2012].

Original: Therefore, any contraction condition derived for the x (or y ) system automatically implies contraction of the other system.
Revised: Therefore, any contraction condition derived for the x (or y ) system automatically implies contraction of the other system.

Original: We exploit this freedom freely throughout the paper.
Revised: We exploit this freedom freely throughout the paper.

Original: A1.2 Contraction Math
Revised: A1.2 Contraction Math

Original: It can be shown that the non-autonomous system

Original: [Equation] t [Equation] is contracting if there exists a metric M ( x , t ) =
Revised: It can be shown that the non-autonomous system˙ x = f ( x , t ) is contracting if there exists a metric M ( x , t ) =

Original: Θ ( x , t ) T Θ ( x , t ) (cid:31) 

Revised: Θ ( x , t ) T Θ ( x , t ) (cid:31)

Original: such that uniformly
Revised: such that uniformly

Original: [Equation]
Revised: [Equation]

Original: where J = ∂ f ∂ x and β > 0 .
Revised: where J = ∂ f ∂ x and β > 0 .

Original: For more details see the main reference [Lohmiller and Slotine, 1998].
Revised: For more details see the main reference [Lohmiller and Slotine, 1998].

Original: Similarly, a non-autonomous discrete-time system
Revised: Similarly, a non-autonomous discrete-time system

Original: [Equation]
Revised: [Equation]

Original: is contracting if
Revised: is contracting if

Original: [Equation]
Revised: [Equation]

Original: A1.2.1 Feedback and Hierarchical Combinations
Revised: A1.2.1 Feedback and Hierarchical Combinations

Original: Consider two systems, independently contracting in constant metrics M 1 and M 2 , which are combined in feedback:
Revised: Consider two systems, independently contracting in constant metrics M 1 and M 2 , which are combined in feedback:

Original: [Equation]
Revised: [Equation]

Original: If the following relationship between B , G , M 1 , and M 2 is satisﬁed:
Revised: If the following relationship between B , G , M 1 , and M 2 is satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: then the combined system is contracting as well.
Revised: then the combined system is contracting as well.

Original: This may be seen as a special case of the feedbackcombination derived in [Tabareau and Slotine, 2006].
Revised: This may be seen as a special case of the feedbackcombination derived in [Tabareau and Slotine, 2006].

Original: The situation is even simpler for hierarchical combinations.
Revised: The situation is even simpler for hierarchical combinations.

Original: Consider again two systems, independently contracting in some metrics, which arecombined in hierarchy:
Revised: Consider again two systems, independently contracting in some metrics, which arecombined in hierarchy:

Original: [Equation]
Revised: [Equation]

Original: where h ( x , t ) is a function with bounded Jacobian.
Revised: where h ( x , t ) is a function with bounded Jacobian.

Original: Then this combined system is contracting in adiagonal metric, as shown in [Lohmiller and Slotine, 1998].
Revised: Then this combined system is contracting in adiagonal metric, as shown in [Lohmiller and Slotine, 1998].

Original: By recursion, this extends to hierarchiesof arbitrary depth.
Revised: By recursion, this extends to hierarchiesof arbitrary depth.

Original: A Extended Discussion
Revised: A Extended Discussion

Original: Given the paucity of existing theory on modular networks, our novel stability conditions and proof ofconcept combination architectures are a signiﬁcant step in an important new direction.
Revised: Given the paucity of existing theory on modular networks, our novel stability conditions and proof ofconcept combination architectures are a signiﬁcant step in an important new direction.

Original: The “networkof networks" approach is evident in the biological brain, and has seen early practical success inapplications such as AlphaGo.
Revised: The “networkof networks" approach is evident in the biological brain, and has seen early practical success inapplications such as AlphaGo.

Original: There is much evidence this line of questioning will be critical in thefuture, and our work is the ﬁrst on stable modular networks.
Revised: There is much evidence this line of questioning will be critical in thefuture, and our work is the ﬁrst on stable modular networks.

Original: Furthermore, we develop an architecture based on such combinations of “vanilla" RNNs that is bothstable and achieves high performance on benchmark sequence classiﬁcation tasks using few trainableparameters (small particularly for sequential CIFAR10).
Revised: Furthermore, we develop an architecture based on such combinations of “vanilla" RNNs that is bothstable and achieves high performance on benchmark sequence classiﬁcation tasks using few trainableparameters (small particularly for sequential CIFAR10).

Original: When considering just the facts about thenetwork, it really has no business performing anywhere near as well as it does.
Revised: When considering just the facts about thenetwork, it really has no business performing anywhere near as well as it does.

Original: Note also that withoutthe stability condition in place, the network performance indeed drops substantially.
Revised: Note also that withoutthe stability condition in place, the network performance indeed drops substantially.

Original: In order to facilitate the extension of this important line of thinking, we provide additional context onthe limitations of our current approach as well as promising ideas for future directions in this section.
Revised: In order to facilitate the extension of this important line of thinking, we provide additional context onthe limitations of our current approach as well as promising ideas for future directions in this section.

Original: A2.1 Limitations
Revised: A2.1 Limitations

Original: One drawback of our approach is that we parameterize each weight matrix in a special way toguarantee stability.
Revised: One drawback of our approach is that we parameterize each weight matrix in a special way toguarantee stability.

Original: In all cases, this requires us to parameterize matrices as the product of severalother matrices.
Revised: In all cases, this requires us to parameterize matrices as the product of severalother matrices.

Original: Thus, we gain stability at the cost of increasing the number of parameters, which canslow down training.
Revised: Thus, we gain stability at the cost of increasing the number of parameters, which canslow down training.

Original: Another current drawback is that we only consider constant metrics.
Revised: Another current drawback is that we only consider constant metrics.

Original: In theory, contraction metricscan be state-dependent as well as time-varying.
Revised: In theory, contraction metricscan be state-dependent as well as time-varying.

Original: Thus it is possible that we have overly restricted thespace of models we consider.
Revised: Thus it is possible that we have overly restricted thespace of models we consider.

Original: Similarly, negative feedback is not the only way to preserve contractionwhen combining two contracting systems.
Revised: Similarly, negative feedback is not the only way to preserve contractionwhen combining two contracting systems.

Original: There are known small-gain theorems in the contractionanalysis literature which accomplish the same task [Slotine, 2003].
Revised: There are known small-gain theorems in the contractionanalysis literature which accomplish the same task [Slotine, 2003].

Original: However, parameterizing theseconditions is less straightforward than parameterizing the negative-feedback condition.
Revised: However, parameterizing theseconditions is less straightforward than parameterizing the negative-feedback condition.

Original: A third limitation of the present work is that it does not give a recipe on how to incorporate anatomicalknowledge into the building of ‘RNNs of RNNs’.
Revised: A third limitation of the present work is that it does not give a recipe on how to incorporate anatomicalknowledge into the building of ‘RNNs of RNNs’.

Original: Our current approach is ‘bottom up’, in the sensethat we describe complicated networks which can be built from simpler ones while ensuring stabilityat every level of construction.
Revised: Our current approach is ‘bottom up’, in the sensethat we describe complicated networks which can be built from simpler ones while ensuring stabilityat every level of construction.

Original: However, for building biological models of the brain it is important toincorporate known anatomical detail (i.e V4 projects to PFC, PFC projects back to V4, etc).
Revised: However, for building biological models of the brain it is important toincorporate known anatomical detail (i.e V4 projects to PFC, PFC projects back to V4, etc).

Original: How todo this in a way that preserves stability is an open and interesting question.
Revised: How todo this in a way that preserves stability is an open and interesting question.

Original: Lastly, we only tested our networks on sequential image classiﬁcation benchmarks.
Revised: Lastly, we only tested our networks on sequential image classiﬁcation benchmarks.

Original: Future work willinclude other benchmarks such as character or word-level language modeling.
Revised: Future work willinclude other benchmarks such as character or word-level language modeling.

Original: Additionally, while weconducted preliminary experiments exploring the role of scale (i.e number of subnetwork RNNs),we did not pursue this at the sizes reached by many modern deep learning applications.
Revised: Additionally, while weconducted preliminary experiments exploring the role of scale (i.e number of subnetwork RNNs),we did not pursue this at the sizes reached by many modern deep learning applications.

Original: Thus it iscurrently unclear if the performance of these stability-constrained models will scale well enough withthe number of subnetworks (or the number of neurons per subnetwork).
Revised: Thus it iscurrently unclear if the performance of these stability-constrained models will scale well enough withthe number of subnetworks (or the number of neurons per subnetwork).

Original: Testing this correctly willrequire extensive experimentation with the various initialization and training settings.
Revised: Testing this correctly willrequire extensive experimentation with the various initialization and training settings.

Original: A2.2 Future Directions
Revised: A2.2 Future Directions

Original: There are numerous future directions enabled by this work.
Revised: There are numerous future directions enabled by this work.

Original: For example, Theorem 6 suggests thata less restrictive contraction condition on W in terms of the eigenvalues of the symmetric part is possible and desirable.
Revised: For example, Theorem 6 suggests thata less restrictive contraction condition on W in terms of the eigenvalues of the symmetric part is possible and desirable.

Original: Meanwhile, Theorem 7 provides important groundwork in ﬁnding such acondition, as it shows the need for a time-varying metric.
Revised: Meanwhile, Theorem 7 provides important groundwork in ﬁnding such acondition, as it shows the need for a time-varying metric.

Original: Investigation of input-dependent metricscould be a fruitful next line of research, and would have far-reaching implications in disciplines suchas curriculum learning.
Revised: Investigation of input-dependent metricscould be a fruitful next line of research, and would have far-reaching implications in disciplines suchas curriculum learning.

Original: Furthermore, the beneﬁcial impact of sparsity on training these stable models suggests a potentialavenue for additional experimental work – in particular adding a regularizing sparsity term duringtraining.
Revised: Furthermore, the beneﬁcial impact of sparsity on training these stable models suggests a potentialavenue for additional experimental work – in particular adding a regularizing sparsity term duringtraining.

Original: This could allow Sparse Combo Net to have its internal subnetwork weights trained withoutlosing the stability guarantee, and conversely it could allow SVD Combo Net to reap some of theperformance beneﬁts of Sparse Combo Net without giving up the ﬂexibility allowed by training saidinternal weights.
Revised: This could allow Sparse Combo Net to have its internal subnetwork weights trained withoutlosing the stability guarantee, and conversely it could allow SVD Combo Net to reap some of theperformance beneﬁts of Sparse Combo Net without giving up the ﬂexibility allowed by training saidinternal weights.

Original: As described in the limitations above, a major experimental next step will be to test our architecturesat greater scale and on more difﬁcult tasks.
Revised: As described in the limitations above, a major experimental next step will be to test our architecturesat greater scale and on more difﬁcult tasks.

Original: Since ‘network of network’ approaches are becomingincreasingly popular, our methodology is relevant to a variety of task types, including reinforcementlearning applications.
Revised: Since ‘network of network’ approaches are becomingincreasingly popular, our methodology is relevant to a variety of task types, including reinforcementlearning applications.

Original: Given the biological inspiration, multi-modal learning tasks may also be ofparticular relevance.
Revised: Given the biological inspiration, multi-modal learning tasks may also be ofparticular relevance.

Original: A Proofs for Main Results
Revised: A Proofs for Main ResultsA3.1 Proof of Feedback Combination Property

Original: A3.1 Proof of Feedback Combination Property

Original: Here we apply existing contraction analysis results to derive equation (3).
Revised: Here we apply existing contraction analysis results to derive equation (3).

Original: Because (3) is aparameterization of a known contraction conditions [Slotine, 2003], we provide the followingstatement in the form of a corollary.
Revised: Because (3) is aparameterization of a known contraction conditions [Slotine, 2003], we provide the followingstatement in the form of a corollary.

Original: Corollary 1 (Network of Networks) .
Revised: Corollary 1 (Network of Networks) .

Original: Consider a collection of p subnetwork RNNs governedby (1) .
Revised: Consider a collection of p subnetwork RNNs governedby (1) .

Original: Assume that these RNNs each have hidden-to-hidden weight matrices { W 1 , . . .
Revised: Assume that these RNNs each have hidden-to-hidden weight matrices { W 1 , . . .

Original: , W p }and are independently contracting in metrics { M 1 , . . .
Revised: , W p }and are independently contracting in metrics { M 1 , . . .

Original: , M p } .
Revised: , M p } .

Original: Deﬁne the block matrices˜ W ≡BlockDiag
Revised: Deﬁne the block matrices ˜ W ≡

Original:  ( W 1 , . . . , W p ) , ˜ M ≡ BlockDiag ( M 1 , . . .
Revised: BlockDiag ( W 1 , . . . , W p ) and ˜ M ≡ BlockDiag ( M 1 , . . .

Original: , M p ) ,the overall state vector ˜ x T ≡( x T 1 · · · xTp ) , and ﬁnally ˜ u T ≡ ( u T 1 · · · u Tp ) 
Revised: , M p ) , as well as the overall state vector˜ x T ≡ ( x T 1 · · · x T 2 ) .

Original: Then the following ‘network of networks’ is globallycontracting in metric ˜ M :
Revised: Then the following ‘network of networks’ is globally contracting in metric ˜ M :

Original: [Equation]
Revised: [Equation]

Original: Where B is an arbitrary square matrix.
Revised: Where B is an arbitrary square matrix.

Original: The matrix L is made up of many individual sub-matrices(the L ij terms in (2) ) which deﬁne the interareal connectivity of the overall network.

Original: Proof.
Revised: Proof.

Original: Consider the differential Lyapunov function:
Revised: Consider the differential Lyapunov function:

Original: [Equation]
Revised: [Equation]

Original: The time-derivative of this function is:
Revised: The time-derivative of this function is:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Since we assume that the RNNs are contracting in isolation, the ﬁrst term in this sum is less that theslowest contracting rate of the individual RNNs, which we call λ > 0 .
Revised: Since we assume that the RNNs are contracting in isolation, the ﬁrst term in this sum is less that theslowest contracting rate of the individual RNNs, which we call λ > 0 .

Original: The second term in the sum isthe zero matrix, by construction.
Revised: The second term in the sum isthe zero matrix, by construction.

Original: Thus the time-derivative of V is upper-bounded by:
Revised: Thus the time-derivative of V is upper-bounded by:

Original: [Equation]
Revised: [Equation]

Original: This implies that the network of networks is contracting with rate λ .
Revised: This implies that the network of networks is contracting with rate λ .

Original: A3.2 Proof of Theorem 1
Revised: A3.2 Proof of Theorem 1

Original: Our ﬁrst theorem is motivated by the observation that if the y-system (described in Section A1.1) is tobe interpreted as a vector of ﬁring rates, it must stay positive for all time.
Revised: Our ﬁrst theorem is motivated by the observation that if the y-system (described in Section A1.1) is tobe interpreted as a vector of ﬁring rates, it must stay positive for all time.

Original: For a linear, time-invariantsystem with positive states, diagonal stability is equivalent to stability.
Revised: For a linear, time-invariantsystem with positive states, diagonal stability is equivalent to stability.

Original: Therefore a natural question isif diagonal stability of a linearized y-system implies anything about stability of the nonlinear system.
Revised: Therefore a natural question isif diagonal stability of a linearized y-system implies anything about stability of the nonlinear system.

Original: More formally, given an excitatory neural network (i.e ∀ ij, W ij ≥ 0 ), if the linear system
Revised: More formally, given an excitatory neural network (i.e ∀ ij, W ij ≥ 0 ), if the linear system

Original: [Equation]
Revised: [Equation]

Original: is stable, then there exists a positive diagonal matrix P such that:
Revised: is stable, then there exists a positive diagonal matrix P such that:

Original: [Equation]
Revised: [Equation]

Original: The following theorem shows that the nonlinear system (1) is indeed contracting in metric P , andextends this result to a more general W by considering only the magnitudes of the weights.
Revised: The following theorem shows that the nonlinear system (1) is indeed contracting in metric P , andextends this result to a more general W by considering only the magnitudes of the weights.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let | W | denote the matrix formed by taking the element-wise absolute value of W .
Revised: Let | W | denote the matrix formed by taking the element-wise absolute value of W .

Original: Ifthere exists a positive, diagonal P such that:
Revised: Ifthere exists a positive, diagonal P such that:

Original: [Equation]
Revised: [Equation]

Original: then (1) is contracting in metric P .
Revised: then (1) is contracting in metric P .

Original: Moreover, if W ii ≤ 0 , then | W | ii may be set to zero to reduceconservatism.
Revised: Moreover, if W ii ≤ 0 , then | W | ii may be set to zero to reduceconservatism.

Original: This condition is particularly straightforward in the common special case where the network does nothave any self weights, with the leak term driving stability.
Revised: This condition is particularly straightforward in the common special case where the network does nothave any self weights, with the leak term driving stability.

Original: While it can be applied to a more general
Revised: While it can be applied to a more general

Original: W , the condition will of course not be met if the network was relying on highly negative values onthe diagonal of W for linear stability.
Revised: W , the condition will of course not be met if the network was relying on highly negative values onthe diagonal of W for linear stability.

Original: As demonstrated by counterexample in the proof of Theorem1, it can be impossible to use the same metric P for the nonlinear RNN in such cases.
Revised: As demonstrated by counterexample in the proof of Theorem1, it can be impossible to use the same metric P for the nonlinear RNN in such cases.

Original: Theorem 1 allows many weight matrices with low magnitudes or a generally sparse structure to beveriﬁed as contracting in the nonlinear system (1), by simply checking a linear stability condition(as linear stability is equivalent to diagonal stability for Metzler matrices too [Narendra and Shorten,2010]).
Revised: Theorem 1 allows many weight matrices with low magnitudes or a generally sparse structure to beveriﬁed as contracting in the nonlinear system (1), by simply checking a linear stability condition(as linear stability is equivalent to diagonal stability for Metzler matrices too [Narendra and Shorten,2010]).

Original: Beyond verifying contraction, Theorem 1 actually provides a metric, with little need for additionalcomputation.
Revised: Beyond verifying contraction, Theorem 1 actually provides a metric, with little need for additionalcomputation.

Original: Not only is it of inherent interest that the same metric can be shared across systems
Revised: Not only is it of inherent interest that the same metric can be shared across systems

Original: inthis case, it is also of use in machine learning applications, where stability certiﬁcates are becomingincreasingly necessary.
Revised: inthis case, it is also of use in machine learning applications, where stability certiﬁcates are becomingincreasingly necessary.

Original: Critically, it is feasible to enforce the condition during training via L2regularization on W .
Revised: Critically, it is feasible to enforce the condition during training via L2regularization on W .

Original: More generally, there are a variety of systems of interest that meet thiscondition but do not meet the well-known maximum singular value condition, including those with ahierarchical structure.
Revised: More generally, there are a variety of systems of interest that meet thiscondition but do not meet the well-known maximum singular value condition, including those with ahierarchical structure.

Original: Proof.
Revised: Proof.

Original: Consider the differential, quadratic Lyapunov function:
Revised: Consider the differential, quadratic Lyapunov function:

Original: [Equation]
Revised: [Equation]

Original: where P (cid:31) 0 is diagonal.
Revised: where P (cid:31) 0 is diagonal.

Original: The time derivative of V is:
Revised: The time derivative of V is:

Original: [Equation]
Revised: [Equation]

Original: where D is a diagonal matrix such that D ii
Revised: where D is a diagonal matrix such that D ii

Original: = dφ i dx ≥ 0 .
Revised: = dφ i dx ≥ 0 .

Original: We can upper bound the quadratic form onthe right as follows:
Revised: We can upper bound the quadratic form onthe right as follows:

Original: [Equation]
Revised: [Equation]

Original: If W ii ≤ 0 , the term P i W ii D i | δx i | 2 contributes non-positively to the overall sum, and can thereforebe set to zero without disrupting the inequality.
Revised: If W ii ≤ 0 , the term P i W ii D i | δx i | 2 contributes non-positively to the overall sum, and can thereforebe set to zero without disrupting the inequality.

Original: Now using the fact that P is positive and diagonal,and therefore δ x T P δ x = | δ x | T P | δ x | , we can upper bound ˙ V as:
Revised: Now using the fact that P is positive and diagonal,and therefore δ x T P δ x = | δ x | T P | δ x | , we can upper bound ˙ V as:

Original: [Equation]
Revised: [Equation]

Original: where | W | ij = | W ij | , and | W | ii = 0 if W ii ≤ 0 and | W | ii = | W ii | if W ii > 0 .
Revised: where | W | ij = | W ij | , and | W | ii = 0 if W ii ≤ 0 and | W | ii = | W ii | if W ii > 0 .

Original: This completes theproof.
Revised: This completes theproof.

Original: Note that W − I is Metzler, and therefore will be Hurwitz stable if and only if P exists [Narendraand Shorten, 2010].
Revised: Note that W − I is Metzler, and therefore will be Hurwitz stable if and only if P exists [Narendraand Shorten, 2010].

Original: It is also worth noting that highly negative diagonal values in W will prevent the same metric Pfrom being used for the nonlinear system.
Revised: It is also worth noting that highly negative diagonal values in W will prevent the same metric Pfrom being used for the nonlinear system.

Original: Therefore the method used in this proof cannot feasibly beadapted to further relax the treatment of the diagonal part of W .
Revised: Therefore the method used in this proof cannot feasibly beadapted to further relax the treatment of the diagonal part of W .

Original: The intuitive reason behind this is that in the symmetric part of the Jacobian, PWD + DW T P− P ,the diagonal self weights will also be scaled down by small D , while the leak portion − P remainsuntouched by D .
Revised: The intuitive reason behind this is that in the symmetric part of the Jacobian, PWD + DW T P− P ,the diagonal self weights will also be scaled down by small D , while the leak portion − P remainsuntouched by D .

Original: Now we actually demonstrate a counterexample, presenting a 2 × 2 symmetric Metzler matrix Wthat is contracting in the identity in the linear system, but cannot be contracting in the identity in thenonlinear system (1):
Revised: Now we actually demonstrate a counterexample, presenting a 2 × 2 symmetric Metzler matrix Wthat is contracting in the identity in the linear system, but cannot be contracting in the identity in thenonlinear system (1):

Original: [Equation]
Revised: [Equation]

Original: To see that it is not possible for the more general nonlinear system with these weights to be contractingin the identity, take D =
Revised: To see that it is not possible for the more general nonlinear system with these weights to be contractingin the identity, take D =

Original: (cid:20) 00(cid:21) .
Revised: (cid:20) 00(cid:21) .

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: −1 .1 .−(cid:21) which has a positive eigenvalue of 14 .
Revised: −1 .1 .−(cid:21) which has a positive eigenvalue of 14 .

Original: A3.3 Proof of Theorem 
Revised: A3.3 Proof of Theorem 2

Original: While regularization may push networks towards satisfying Theorem 1, strictly enforcing thecondition during optimization is not straightforward.
Revised: While regularization may push networks towards satisfying Theorem 1, strictly enforcing thecondition during optimization is not straightforward.

Original: This motivates the rest of our theorems, whichderive contraction results for specially structured weight matrices.
Revised: This motivates the rest of our theorems, whichderive contraction results for specially structured weight matrices.

Original: Unlike Theorem 1, these resultshave direct parameterizations which can easily be plugged into modern optimization libraries.
Revised: Unlike Theorem 1, these resultshave direct parameterizations which can easily be plugged into modern optimization libraries.

Original: Theorem 2.
Revised: Theorem 2.

Original: If W = W T and g W ≺ I , then (1) is contracting.
Revised: If W = W T and g W ≺ I , then (1) is contracting.

Original: When W is symmetric, (1) may be seen as a continuous-time Hopﬁeld network.
Revised: When W is symmetric, (1) may be seen as a continuous-time Hopﬁeld network.

Original: Continuous-time
Revised: Continuous-time

Original: Hopﬁeld networks with symmetric weights were recently shown to be closely related to Transformerarchitectures [Krotov and Hopﬁeld, 2020, Ramsauer et al., 2020].
Revised: Hopﬁeld networks with symmetric weights were recently shown to be closely related to Transformerarchitectures [Krotov and Hopﬁeld, 2020, Ramsauer et al., 2020].

Original: Speciﬁcally, the dot-productattention rule may be seen as a discretization of the continuous-time Hopﬁeld network with softmaxactivation function [Krotov and Hopﬁeld, 2020].
Revised: Speciﬁcally, the dot-productattention rule may be seen as a discretization of the continuous-time Hopﬁeld network with softmaxactivation function [Krotov and Hopﬁeld, 2020].

Original: Our results here provide a simple sufﬁcient (andnearly necessary, see above remark) condition for global exponential stability of a given trajectory forthe Hopﬁeld network.
Revised: Our results here provide a simple sufﬁcient (andnearly necessary, see above remark) condition for global exponential stability of a given trajectory forthe Hopﬁeld network.

Original: In the case where the input into the network is constant, this trajectory is a ﬁxedpoint.
Revised: In the case where the input into the network is constant, this trajectory is a ﬁxedpoint.

Original: Moreover, each trajectory associated with a unique input is guaranteed to be unique.
Revised: Moreover, each trajectory associated with a unique input is guaranteed to be unique.

Original: Finally,we note that our results are ﬂexible with respect to activation functions so long as they satisfy theslope-restriction condition.
Revised: Finally,we note that our results are ﬂexible with respect to activation functions so long as they satisfy theslope-restriction condition.

Original: This ﬂexibility may be useful when, for example, considering recent workshowing that standard activation functions may be advantageously replaced by attention mechanisms[Dai
Revised: This ﬂexibility may be useful when, for example, considering recent workshowing that standard activation functions may be advantageously replaced by attention mechanisms[Dai

Original: et al., 2020].
Revised: et al., 2020].

Original: Proof.
Revised: Proof.

Original: We begin by writing W = R − P for some unknown R = R T and P = P T (cid:31) 0 .
Revised: We begin by writing W = R − P for some unknown R = R T and P = P T (cid:31) 0 .

Original: Theapproach of this proof is to show by construction that the condition g W ≺ I implies the existence ofan R and P such that the system is contracting in metric P .
Revised: Theapproach of this proof is to show by construction that the condition g W ≺ I implies the existence ofan R and P such that the system is contracting in metric P .

Original: We consider the y version of the RNN,which as discussed above is equivalent to the x version via an afﬁne transformation.
Revised: We consider the y version of the RNN,which as discussed above is equivalent to the x version via an afﬁne transformation.

Original: Consider the contraction condition:
Revised: Consider the contraction condition:

Original: [Equation]
Revised: [Equation]

Original: with β > 0 .
Revised: with β > 0 .

Original: Substituting in the deﬁnitions of W and M , this condition becomes:
Revised: Substituting in the deﬁnitions of W and M , this condition becomes:

Original: [Equation]
Revised: [Equation]

Original: Simplifying the terms and collecting them all on one side, the above may be written as:
Revised: Simplifying the terms and collecting them all on one side, the above may be written as:

Original: [Equation]
Revised: [Equation]

Original: via the Schur complement, the above term will be satisﬁed if:
Revised: via the Schur complement, the above term will be satisﬁed if:

Original: [Equation]
Revised: [Equation]

Original: We continue by setting P = γ 2 RR with γ 

Revised: We continue by setting P = γ 2 RR with γ

Original: = g 2(

Revised: = g 2(

Original: − β ) , so that the above inequality is satisﬁed.
Revised: − β ) , so that the above inequality is satisﬁed.

Original: Atthis point, we have shown that if W can be written as:
Revised: Atthis point, we have shown that if W can be written as:

Original: [Equation]
Revised: [Equation]

Original: then (1) is contracting in metric M = γ 2 RR .
Revised: then (1) is contracting in metric M = γ 2 RR .

Original: What remains to be shown is that if the condition:
Revised: What remains to be shown is that if the condition:

Original: [Equation]
Revised: [Equation]

Original: Is satisﬁed, then this implies the existence of an R such that the above is true.
Revised: Is satisﬁed, then this implies the existence of an R such that the above is true.

Original: To show that this isindeed the case, assume that:
Revised: To show that this isindeed the case, assume that:

Original: [Equation]
Revised: [Equation]

Original: Substituting in the deﬁnition of γ , this is just the statement that:
Revised: Substituting in the deﬁnition of γ , this is just the statement that:

Original: [Equation]
Revised: [Equation]

Original: Setting β
Revised: Setting β

Original: = 2 λ > 0 , this yields:
Revised: = 2 λ > 0 , this yields:

Original: [Equation]
Revised: [Equation]

Original: Since W is orthogonal, we have the eigendecomposition:
Revised: Since W is orthogonal, we have the eigendecomposition:

Original: [Equation]
Revised: [Equation]

Original: where V T V =
Revised: where V T V =

Original: I and Λ is a diagonal matrix containing the eigenvalues of W .
Revised: I and Λ is a diagonal matrix containing the eigenvalues of W .

Original: Denote the symmetricsquare-root of this expression as S :
Revised: Denote the symmetricsquare-root of this expression as S :

Original: [Equation]
Revised: [Equation]

Original: Which implies that:
Revised: Which implies that:

Original: [Equation]
Revised: [Equation]

Original: We now deﬁne R in terms of S as follows:
Revised: We now deﬁne R in terms of S as follows:

Original: [Equation]
Revised: [Equation]

Original: Which means that:
Revised: Which means that:

Original: [Equation]
Revised: [Equation]

Original: Expanding out the right side, we get:
Revised: Expanding out the right side, we get:

Original: [Equation]
Revised: [Equation]

Original: Subtracting 14 γ 2 I from both sides yields:
Revised: Subtracting 14 γ 2 I from both sides yields:

Original: [Equation]
Revised: [Equation]

Original: As desired.
Revised: As desired.

Original: A3.4 Proof of Theorem 3
Revised: A3.4 Proof of Theorem 3

Original: Theorem 3.
Revised: Theorem 3.

Original: If there exists positive diagonal matrices P 1 and P 2 , as well as Q = Q T (cid:31) 0 such that
Revised: If there exists positive diagonal matrices P 1 and P 2 , as well as Q = Q T (cid:31) 0 such that

Original: [-W = − P 
-]
Revised: [-W = − P 
-][Equation]

Original:  QPthen (1) is contracting in metric M =
Revised: = − P 1 QPthen (1) is contracting in metric M =

Original: ( P 1 QP 1 ) − 1 .
Revised: ( P 1 QP 1 ) − 1 .

Original: Proof.
Revised: Proof.

Original: Consider again a differential Lyapunov function:
Revised: Consider again a differential Lyapunov function:

Original: [Equation]
Revised: [Equation]

Original: the time derivative is equal to:
Revised: the time derivative is equal to:

Original: [Equation]
Revised: [Equation]

Original: Substituting in the deﬁnitions of W and M , we get:
Revised: Substituting in the deﬁnitions of W and M , we get:

Original: [Equation]
Revised: [Equation]

Original: Therefore V converges exponentially to zero.
Revised: Therefore V converges exponentially to zero.

Original: A3.5 Proof of Theorem 4
Revised: A3.5 Proof of Theorem 4

Original: Theorem 4.
Revised: Theorem 4.

Original: If g W − I is triangular and Hurwitz, then (1) is contracting in a diagonal metric.
Revised: If g W − I is triangular and Hurwitz, then (1) is contracting in a diagonal metric.

Original: Note that in the case of a triangular weight matrix, the system (1) may be seen as a feedforward (i.ehierarchical) network.
Revised: Note that in the case of a triangular weight matrix, the system (1) may be seen as a feedforward (i.ehierarchical) network.

Original: Therefore, this result follows from the combination properties of contractingsystems.
Revised: Therefore, this result follows from the combination properties of contractingsystems.

Original: However, our proof provides a means of explicitly ﬁnding a metric for this system.
Revised: However, our proof provides a means of explicitly ﬁnding a metric for this system.

Original: Proof.
Revised: Proof.

Original: Without loss of generality, assume that W is lower triangular.
Revised: Without loss of generality, assume that W is lower triangular.

Original: This implies that W ij = 0 ifi ≤ j .
Revised: This implies that W ij = 0 ifi ≤ j .

Original: Now consider the generalized Jacobian:
Revised: Now consider the generalized Jacobian:

Original: [Equation]
Revised: [Equation]

Original: with Γ diagonal and Γ i = (cid:15) i where (cid:15) > 0 .
Revised: with Γ diagonal and Γ i = (cid:15) i where (cid:15) > 0 .

Original: Because Γ is diagonal, the generalized Jacobian is equalto:
Revised: Because Γ is diagonal, the generalized Jacobian is equalto:

Original: [Equation]
Revised: [Equation]

Original: Now note that:
Revised: Now note that:

Original: [Equation]
Revised: [Equation]

Original: Where i ≤ j , we have W ij = 0 by assumption.
Revised: Where i ≤ j , we have W ij = 0 by assumption.

Original: Therefore, the only nonzero entries are where i ≥ j .
Revised: Therefore, the only nonzero entries are where i ≥ j .

Original: This means that by making (cid:15) arbitrarily small, we can make ΓWΓ − 1 approach a diagonal matrixwith W ii along the diagonal.
Revised: This means that by making (cid:15) arbitrarily small, we can make ΓWΓ − 1 approach a diagonal matrixwith W ii along the diagonal.

Original: Therefore, if:
Revised: Therefore, if:

Original: [Equation]
Revised: [Equation]

Original: the nonlinear system is contracting.
Revised: the nonlinear system is contracting.

Original: Since W is triangular, W ii are the eigenvalues of W , meaningthat this condition is equivalent to g W − I being Hurwitz.
Revised: Since W is triangular, W ii are the eigenvalues of W , meaningthat this condition is equivalent to g W − I being Hurwitz.

Original: A3.6 Proof of Theorem5
Revised: A3.6 Proof of Theorem

Original: Theorem 5.
Revised: Theorem 5.

Original: If there exists a positive diagonal matrix P such that:
Revised: If there exists a positive diagonal matrix P such that:

Original: [Equation]
Revised: [Equation]

Original: then (1) is contracting in metric P .
Revised: then (1) is contracting in metric P .

Original: Note that this is equivalent to the discrete-time diagonal stability condition developed in [Revay and
Revised: Note that this is equivalent to the discrete-time diagonal stability condition developed in [Revay and

Original: Manchester, 2020], for a constant metric.
Revised: Manchester, 2020], for a constant metric.

Original: Note also that when M = I , Theorem 5 is identical tochecking the maximum singular value of W , a previously established condition for stability of (1).
Revised: Note also that when M = I , Theorem 5 is identical tochecking the maximum singular value of W , a previously established condition for stability of (1).

Original: However a much larger set of weight matrices are found via the condition when M = P instead.
Revised: However a much larger set of weight matrices are found via the condition when M = P instead.

Original: Proof.
Revised: Proof.

Original: Consider the generalized Jacobian:
Revised: Consider the generalized Jacobian:

Original: [Equation]
Revised: [Equation]

Original: where D is a diagonal matrix with D ii
Revised: where D is a diagonal matrix with D ii

Original: = dφ i dx i .
Revised: = dφ i dx i .

Original: Using the subadditivity of the matrix measure µ 2 ofthe generalized Jacobian we get:
Revised: Using the subadditivity of the matrix measure µ 2 ofthe generalized Jacobian we get:

Original: [Equation]
Revised: [Equation]

Original: Now using the fact that µ 2 ( · ) ≤ || · || 2 we have:
Revised: Now using the fact that µ 2 ( · ) ≤ || · || 2 we have:

Original: [Equation]
Revised: [Equation]

Original: Using the deﬁnition of the 2-norm, imposing the condition µ 2 ( F ) ≤ 0 may be written:
Revised: Using the deﬁnition of the 2-norm, imposing the condition µ 2 ( F ) ≤ 0 may be written:

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: A3.7 Proof of Theorem 6
Revised: A3.7 Proof of Theorem 6

Original: Theorem 6.
Revised: Theorem 6.

Original: Let D be a positive, diagonal matrix with D ii
Revised: Let D be a positive, diagonal matrix with D ii

Original: = dφ i dx i , and let P be an arbitrary, positivediagonal matrix.
Revised: = dφ i dx i , and let P be an arbitrary, positivediagonal matrix.

Original: If:
Revised: If:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: for c, β > 0 , then (1) is contracting in metric D with rate β .
Revised: for c, β > 0 , then (1) is contracting in metric D with rate β .

Original: Proof.
Revised: Proof.

Original: Consider the differential, quadratic Lyapunov function:
Revised: Consider the differential, quadratic Lyapunov function:

Original: [Equation]
Revised: [Equation]

Original: where D (cid:31) 0 is as deﬁned above.
Revised: where D (cid:31) 0 is as deﬁned above.

Original: The time derivative of V is:
Revised: The time derivative of V is:

Original: [Equation]
Revised: [Equation]

Original: The second term on the right can be factored as:
Revised: The second term on the right can be factored as:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality was obtained by substituting in the ﬁrst assumption above.
Revised: where the last inequality was obtained by substituting in the ﬁrst assumption above.

Original: Combining thiswith the expression for ˙ V , we have:
Revised: Combining thiswith the expression for ˙ V , we have:

Original: [Equation]
Revised: [Equation]

Original: Substituting in the second assumption, we have:
Revised: Substituting in the second assumption, we have:

Original: [Equation]
Revised: [Equation]

Original: and thus V converges exponentially to 0 with rate β .
Revised: and thus V converges exponentially to 0 with rate β .

Original: A3.8 Proof of Theorem 7
Revised: A3.8 Proof of Theorem 7

Original: Theorem 7.
Revised: Theorem 7.

Original: Satisfaction of the condition
Revised: Satisfaction of the condition

Original: [Equation]
Revised: [Equation]

Original: is NOT sufﬁcient to show global contraction of the general nonlinear RNN (1) in any constant metric.
Revised: is NOT sufﬁcient to show global contraction of the general nonlinear RNN (1) in any constant metric.

Original: High levels of antisymmetry in W can make it impossible to ﬁnd such a metric, which we demonstratevia a 2 × 2 counterexample of the form
Revised: High levels of antisymmetry in W can make it impossible to ﬁnd such a metric, which we demonstratevia a 2 × 2 counterexample of the form

Original: [Equation]
Revised: [Equation]

Original: with c ≥ 2 .
Revised: with c ≥ 2 .

Original: Note that g W sym − I = g W + W T− I ≺ 0 is equivalent to the condition for contraction of thesystem with linear activation in the identity metric.
Revised: Note that g W sym − I = g W + W T− I ≺ 0 is equivalent to the condition for contraction of thesystem with linear activation in the identity metric.

Original: The main intuition behind this counterexample is that high levels of antisymmetry can prevent aconstant metric from being found in the nonlinear system.
Revised: The main intuition behind this counterexample is that high levels of antisymmetry can prevent aconstant metric from being found in the nonlinear system.

Original: This is because D is a diagonal matrixwith values between 0 and 1, so the primary functionality it can have in the symmetric part of the
Revised: This is because D is a diagonal matrixwith values between 0 and 1, so the primary functionality it can have in the symmetric part of the

Original: Jacobian is to downweight the outputs of certain neurons selectively.
Revised: Jacobian is to downweight the outputs of certain neurons selectively.

Original: In the extreme case of all 0 or 1values, we can think of this as selecting a subnetwork of the original network, and taking each ofthe remaining neurons to be single unit systems receiving input from the subnetwork.
Revised: In the extreme case of all 0 or 1values, we can think of this as selecting a subnetwork of the original network, and taking each ofthe remaining neurons to be single unit systems receiving input from the subnetwork.

Original: For a givenstatic conﬁguration of D (think linear gains), this is a hierarchical system that will be stable if thesubnetwork is stable.
Revised: For a givenstatic conﬁguration of D (think linear gains), this is a hierarchical system that will be stable if thesubnetwork is stable.

Original: But as D can evolve over time when a nonlinearity is introduced, we wouldneed to ﬁnd a constant metric that can serve completely distinct hierarchical structures simultaneously-
Revised: But as D can evolve over time when a nonlinearity is introduced, we wouldneed to ﬁnd a constant metric that can serve completely distinct hierarchical structures simultaneously-

Original: which is not always possible.
Revised: which is not always possible.

Original: Put in terms of matrix algebra, D can zero out columns of W , but not their corresponding rows.
Revised: Put in terms of matrix algebra, D can zero out columns of W , but not their corresponding rows.

Original: Sofor a given weight pair w ij , w ji , which has entry in W sym = w ij + w ji, if D i = 0 and D j = 1 , thei, j entry in ( WD ) sym will be guaranteed to have lower magnitude if the signs of w ij and w ji arethe same, but guaranteed to have higher magnitude if the signs are different.
Revised: Sofor a given weight pair w ij , w ji , which has entry in W sym = w ij + w ji, if D i = 0 and D j = 1 , thei, j entry in ( WD ) sym will be guaranteed to have lower magnitude if the signs of w ij and w ji arethe same, but guaranteed to have higher magnitude if the signs are different.

Original: Thus if the linear systemwould be stable based on magnitudes alone D poses no real threat, but if the linear system requiresantisymmetry to be stable, D can make proving contraction quite complicated (if possible at all).
Revised: Thus if the linear systemwould be stable based on magnitudes alone D poses no real threat, but if the linear system requiresantisymmetry to be stable, D can make proving contraction quite complicated (if possible at all).

Original: Proof.
Revised: Proof.

Original: The nonlinear system is globally contracting in a constant metric if there exists a symmetric,positive deﬁnite M such that the symmetric part of the Jacobian for the system, ( MWD ) sym − Mis negative deﬁnite uniformly.
Revised: The nonlinear system is globally contracting in a constant metric if there exists a symmetric,positive deﬁnite M such that the symmetric part of the Jacobian for the system, ( MWD ) sym − Mis negative deﬁnite uniformly.

Original: Therefore ( MWD ) sym − M ≺ 0 must hold for all possible D if Mis a constant metric the system globally contracts in with any allowed activation function, as somecombination of settings to obtain a particular D can always be found.
Revised: Therefore ( MWD ) sym − M ≺ 0 must hold for all possible D if Mis a constant metric the system globally contracts in with any allowed activation function, as somecombination of settings to obtain a particular D can always be found.

Original: Thus to prove the main claim, we present here a simple 2-neuron system that is contracting in theidentity metric with linear activation function, but can be shown to have no M that simultaneouslysatisﬁes the ( MWD ) sym − M ≺ 0 condition for two different possible D matrices.
Revised: Thus to prove the main claim, we present here a simple 2-neuron system that is contracting in theidentity metric with linear activation function, but can be shown to have no M that simultaneouslysatisﬁes the ( MWD ) sym − M ≺ 0 condition for two different possible D matrices.

Original: To begin, take
Revised: To begin, take

Original: W = (cid:20) 

Revised: W = (cid:20)

Original: −  (cid:21)
Revised: −  (cid:21)

Original: Note that any off-diagonal magnitude ≥ 2 would work, as this is the point at which 12 of one of theweights (found in W sym when the other is zeroed) will have magnitude too large for ( WD ) sym − Ito be stable.
Revised: Note that any off-diagonal magnitude ≥ 2 would work, as this is the point at which 12 of one of theweights (found in W sym when the other is zeroed) will have magnitude too large for ( WD ) sym − Ito be stable.

Original: Looking at the linear system, we can see it is contracting in the identity because
Revised: Looking at the linear system, we can see it is contracting in the identity because

Original: [Equation]
Revised: [Equation]

Original: Now consider ( MWD ) sym − M with D taking two possible values of
Revised: Now consider ( MWD ) sym − M with D taking two possible values of

Original: [Equation]
Revised: [Equation]

Original: (cid:21) and
Revised: (cid:21) and

Original: D

Original: [Equation]
Revised: [Equation]

Original: (cid:20)
Revised: (cid:20)

Original: [Equation]
Revised: [Equation]

Original: We want to ﬁnd some symmetric, positive deﬁnite M = (cid:20) a m m b (cid:21) such that ( MWD 1 ) sym − Mand ( MWD 2 ) sym − M are both negative deﬁnite.
Revised: We want to ﬁnd some symmetric, positive deﬁnite M = (cid:20) a m m b (cid:21) such that ( MWD 1 ) sym − Mand ( MWD 2 ) sym − M are both negative deﬁnite.

Original: Working out the matrix multiplication, we get
Revised: Working out the matrix multiplication, we get

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and(

Original: [Equation]
Revised: [Equation]

Original: We can now check necessary conditions for negative deﬁniteness on these two matrices, as well as forpositive deﬁniteness on M , to try to ﬁnd an M that will satisfy all these conditions simultaneously.
Revised: We can now check necessary conditions for negative deﬁniteness on these two matrices, as well as forpositive deﬁniteness on M , to try to ﬁnd an M that will satisfy all these conditions simultaneously.

Original: In this process we will reach a contradiction, showing that no such M can exist.
Revised: In this process we will reach a contradiction, showing that no such M can exist.

Original: A necessary condition for positive deﬁniteness in a real, symmetric n × n matrix X is x ii > 0 , andfor negative deﬁniteness x ii < 0 .
Revised: A necessary condition for positive deﬁniteness in a real, symmetric n × n matrix X is x ii > 0 , andfor negative deﬁniteness x ii < 0 .

Original: Another well known necessary condition for deﬁniteness of a realsymmetric matrix is |
Revised: Another well known necessary condition for deﬁniteness of a realsymmetric matrix is |

Original: [Equation]
Revised: [Equation]

Original: | x ij | ∀ i (cid:54) =
Revised: | x ij | ∀ i (cid:54) =

Original: j .
Revised: j .

Original: See
Revised: See

Original: [Weisstein] for more info onthese conditions.
Revised: [Weisstein] for more info onthese conditions.

Original: Thus we will require a and b to be positive, and can identify the following conditions as necessary forour 3 matrices to all meet the requisite deﬁniteness conditions:
Revised: Thus we will require a and b to be positive, and can identify the following conditions as necessary forour 3 matrices to all meet the requisite deﬁniteness conditions:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the necessary condition for M to be PD, a + b > 2 | m | , is not listed, as it is automaticallysatisﬁed if (6) and (7) are.
Revised: Note that the necessary condition for M to be PD, a + b > 2 | m | , is not listed, as it is automaticallysatisﬁed if (6) and (7) are.

Original: It is easy to see that if m = 0 , conditions (8) and (9) will result in the contradictory conditions a > band b > a respectively, so we will require a metric with off-diagonal elements.
Revised: It is easy to see that if m = 0 , conditions (8) and (9) will result in the contradictory conditions a > band b > a respectively, so we will require a metric with off-diagonal elements.

Original: To make the absolutevalues easier to deal with, we will check m > 0 and m < 0 cases independently.
Revised: To make the absolutevalues easier to deal with, we will check m > 0 and m < 0 cases independently.

Original: First we take m > 0 .
Revised: First we take m > 0 .

Original: By condition (6) we must have a > 2 m , so between that and knowing the signsof all unknowns are positive, we can reduce many of the absolute values.
Revised: By condition (6) we must have a > 2 m , so between that and knowing the signsof all unknowns are positive, we can reduce many of the absolute values.

Original: Condition (8) becomesa + b − 2 m > | 2 b − 2 m | , and condition (9) becomes a + b + 2 m > 2 a + 2 m , which is equivalent to b > a .
Revised: Condition (8) becomesa + b − 2 m > | 2 b − 2 m | , and condition (9) becomes a + b + 2 m > 2 a + 2 m , which is equivalent to b > a .

Original: If b > a we must also have b > m , so condition (8) further reduces to a + b − 2 m > 2 b − 2 m ,which is equivalent to a > b .
Revised: If b > a we must also have b > m , so condition (8) further reduces to a + b − 2 m > 2 b − 2 m ,which is equivalent to a > b .

Original: Therefore we have again reached contradictory conditions.
Revised: Therefore we have again reached contradictory conditions.

Original: A very similar approach can be applied when m < 0 .
Revised: A very similar approach can be applied when m < 0 .

Original: Using condition (7) and the known signs wereduce condition (8) to 2 | m | + a + b > 2 b + 2 | m | , i.e. a > b .
Revised: Using condition (7) and the known signs wereduce condition (8) to 2 | m | + a + b > 2 b + 2 | m | , i.e. a > b .

Original: Meanwhile condition (9) works out toa + b − 2 | m | > 2 a − 2 | m | , i.e. b > a .
Revised: Meanwhile condition (9) works out toa + b − 2 | m | > 2 a − 2 | m | , i.e. b > a .

Original: Therefore it is impossible for a single constant M to accommodate both D 1 and D 2 , so that noconstant metric can exist for W to be contracting in when a nonlinearity is introduced that canpossibly have derivative reaching both of these conﬁgurations.
Revised: Therefore it is impossible for a single constant M to accommodate both D 1 and D 2 , so that noconstant metric can exist for W to be contracting in when a nonlinearity is introduced that canpossibly have derivative reaching both of these conﬁgurations.

Original: One real world example of such anonlinearity is ReLU.
Revised: One real world example of such anonlinearity is ReLU.

Original: Given a sufﬁciently high negative input to one of the units and a sufﬁcientlyhigh positive input to the other, D can reach one of these conﬁgurations.
Revised: Given a sufﬁciently high negative input to one of the units and a sufﬁcientlyhigh positive input to the other, D can reach one of these conﬁgurations.

Original: The targeted inputs couldthen ﬂip at any time to reach the other conﬁguration.
Revised: The targeted inputs couldthen ﬂip at any time to reach the other conﬁguration.

Original: An additional condition we could impose on the activation function is to require it to be a strictlyincreasing function, so that the activation function derivative can never actually reach 0.
Revised: An additional condition we could impose on the activation function is to require it to be a strictlyincreasing function, so that the activation function derivative can never actually reach 0.

Original: We will nowshow that a very similar counterexample applies in this case, by taking
Revised: We will nowshow that a very similar counterexample applies in this case, by taking

Original: D 1 ∗ =
Revised: D 1 ∗ =

Original: (cid:20) 1 (cid:15) (cid:21) and
Revised: (cid:20) 1 (cid:15) (cid:21) and

Original: D 2 ∗ =
Revised: D 2 ∗ =

Original: (cid:20) (cid:15) 0(cid:21)
Revised: (cid:20) (cid:15) 0(cid:21)

Original: Note here that the W used above produced a ( WD ) sym − I that just barely avoided being negativedeﬁnite with the original D 1 and D 2 , so we will have to increase the values on the off-diagonals a bitfor this next example.
Revised: Note here that the W used above produced a ( WD ) sym − I that just barely avoided being negativedeﬁnite with the original D 1 and D 2 , so we will have to increase the values on the off-diagonals a bitfor this next example.

Original: In fact anything with magnitude larger than 2 will have some (cid:15) > 0 that willcause a constant metric to be impossible, but for simplicity we will now take
Revised: In fact anything with magnitude larger than 2 will have some (cid:15) > 0 that willcause a constant metric to be impossible, but for simplicity we will now take

Original: [Equation]
Revised: [Equation]

Original: Note that with W ∗ , even just halving one of the off-diagonals while keeping the other intact willproduce a ( WD ) sym − I that is not negative deﬁnite.
Revised: Note that with W ∗ , even just halving one of the off-diagonals while keeping the other intact willproduce a ( WD ) sym − I that is not negative deﬁnite.

Original: Anything less than halving however willkeep the identity metric valid.
Revised: Anything less than halving however willkeep the identity metric valid.

Original: Therefore, we expect that taking (cid:15) in D 1 ∗ and D 2 ∗ to be in the range0 .
Revised: Therefore, we expect that taking (cid:15) in D 1 ∗ and D 2 ∗ to be in the range0 .

Original: 5 ≥ (cid:15) > 0 will also cause issues when trying to obtain a constant metric.
Revised: 5 ≥ (cid:15) > 0 will also cause issues when trying to obtain a constant metric.

Original: We will now actually show via a similar proof to the above that M is impossible to ﬁnd for W ∗ when(cid:15) ≤ 0 .
Revised: We will now actually show via a similar proof to the above that M is impossible to ﬁnd for W ∗ when(cid:15) ≤ 0 .

Original: This result is compelling because it not only shows that (cid:15) does not need to be a particularlysmall value, but it also drives home the point about antisymmetry - the larger in magnitude theantisymmetric weights are, the larger the (cid:15) where we will begin to encounter problems.
Revised: This result is compelling because it not only shows that (cid:15) does not need to be a particularlysmall value, but it also drives home the point about antisymmetry - the larger in magnitude theantisymmetric weights are, the larger the (cid:15) where we will begin to encounter problems.

Original: Working out the matrix multiplication again, we now get
Revised: Working out the matrix multiplication again, we now get

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Resulting in two new main necessary conditions:
Revised: Resulting in two new main necessary conditions:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As well as new conditions on the diagonal elements:
Revised: As well as new conditions on the diagonal elements:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We will now proceed with trying to ﬁnd a, b, m that can simultaneously meet all conditions, setting(cid:15)
Revised: We will now proceed with trying to ﬁnd a, b, m that can simultaneously meet all conditions, setting(cid:15)

Original: = 0 .
Revised: = 0 .

Original: 5 for simplicity.
Revised: 5 for simplicity.

Original: Looking at m = 0 , we can see again that M will require off-diagonal elements, as condition (10) isnow equivalent to the condition a + b > | 4 b − 2 a | and condition (11) is similarly now equivalent toa + b > | 4 a − 2 b | .
Revised: Looking at m = 0 , we can see again that M will require off-diagonal elements, as condition (10) isnow equivalent to the condition a + b > | 4 b − 2 a | and condition (11) is similarly now equivalent toa + b > | 4 a − 2 b | .

Original: Evaluating these conditions in more detail, if we assume 4 b > 2 a and 4 a > 2 b , we can remove theabsolute value and the conditions work out to the contradicting 3 a > 3 b and 3 b > 3 a respectively.
Revised: Evaluating these conditions in more detail, if we assume 4 b > 2 a and 4 a > 2 b , we can remove theabsolute value and the conditions work out to the contradicting 3 a > 3 b and 3 b > 3 a respectively.

Original: As an aside, if (cid:15) > 0 . 5 , this would no longer be the case, whereas with (cid:15) < 0 .
Revised: As an aside, if (cid:15) > 0 . 5 , this would no longer be the case, whereas with (cid:15) < 0 .

Original: 5 , the conditions wouldbe pushed even further in opposite directions.
Revised: 5 , the conditions wouldbe pushed even further in opposite directions.

Original: If we instead assume 2 a > 4 b , this means 4 a > 2 b , so the latter condition would still lead to b > a ,contradicting the original assumption of 2 a > 4 b .
Revised: If we instead assume 2 a > 4 b , this means 4 a > 2 b , so the latter condition would still lead to b > a ,contradicting the original assumption of 2 a > 4 b .

Original: 2 b > 4 a causes a contradiction analogously.
Revised: 2 b > 4 a causes a contradiction analogously.

Original: Trying 4 b = 2 a will lead to the other condition becoming b > 2 a , once again a contradiction.
Revised: Trying 4 b = 2 a will lead to the other condition becoming b > 2 a , once again a contradiction.

Original: Thus adiagonal M is impossible
Revised: Thus adiagonal M is impossible

Original: So now we again break down the conditions into m > 0 and m < 0 cases, ﬁrst looking at m > 0 .
Revised: So now we again break down the conditions into m > 0 and m < 0 cases, ﬁrst looking at m > 0 .

Original: Using condition (12) and knowing all unknowns have positive sign, condition (10) reduces toa + b − 2 m > | 4 b − 2( a + m ) | and condition (11) reduces to a + b + 2 m > | 4 a − 2( b − m ) | .
Revised: Using condition (12) and knowing all unknowns have positive sign, condition (10) reduces toa + b − 2 m > | 4 b − 2( a + m ) | and condition (11) reduces to a + b + 2 m > | 4 a − 2( b − m ) | .

Original: Thislooks remarkably similar to the m = 0 case, except now condition (10) has − 2 m added to both sides(inside the absolute value), and condition (11) has 2 m added to both sides in the same manner.
Revised: Thislooks remarkably similar to the m = 0 case, except now condition (10) has − 2 m added to both sides(inside the absolute value), and condition (11) has 2 m added to both sides in the same manner.

Original: If4 b > 2( a + m ) the − 2 m term on each side will simply cancel, and similarly if 4 a > 2( b − m ) the+2 m terms will cancel, leaving us with the same contradictory conditions as before.
Revised: If4 b > 2( a + m ) the − 2 m term on each side will simply cancel, and similarly if 4 a > 2( b − m ) the+2 m terms will cancel, leaving us with the same contradictory conditions as before.

Original: Therefore we check 2( a + m ) > 4 b .
Revised: Therefore we check 2( a + m ) > 4 b .

Original: This rearranges to 2 a > 2(2 b − m ) > 2( b − m ) , so that fromcondition (11) we get b > a .
Revised: This rearranges to 2 a > 2(2 b − m ) > 2( b − m ) , so that fromcondition (11) we get b > a .

Original: Subbing condition (12) in to 2( a + m ) > 4 b gives 8 b < 4 a + 4 m < 5 ai.e.
Revised: Subbing condition (12) in to 2( a + m ) > 4 b gives 8 b < 4 a + 4 m < 5 ai.e.

Original: b < 58 a , a contradiction.
Revised: b < 58 a , a contradiction.

Original: The analogous issue arises if trying 2( b − m ) > 4 a .
Revised: The analogous issue arises if trying 2( b − m ) > 4 a .

Original: Trying2( a + m ) =
Revised: Trying2( a + m ) =

Original: 4 b gives m = 2 b − a , which in condition (11) results in 5 b − a > | 6 a − 6 b | , while incondition (12) leads to 5 a > 8 b , so (11) can further reduce to 5 b − a > 6 a − 6 b i.e. 11 b > 7 a .
Revised: 4 b gives m = 2 b − a , which in condition (11) results in 5 b − a > | 6 a − 6 b | , while incondition (12) leads to 5 a > 8 b , so (11) can further reduce to 5 b − a > 6 a − 6 b i.e. 11 b > 7 a .

Original: Butb > 711 a and b < 58 a is a contradiction.
Revised: Butb > 711 a and b < 58 a is a contradiction.

Original: Thus there is no way for m > 0 to work.
Revised: Thus there is no way for m > 0 to work.

Original: Finally, trying m < 0 , we now use condition (13) and the signs of the unknowns to reduce condition(10) to a + b + 2 | m | > | 4 b − 2( a − | m | ) | and condition (11) to a + b − 2 | m | > | 4 a − 2( b + | m | ) | .
Revised: Finally, trying m < 0 , we now use condition (13) and the signs of the unknowns to reduce condition(10) to a + b + 2 | m | > | 4 b − 2( a − | m | ) | and condition (11) to a + b − 2 | m | > | 4 a − 2( b + | m | ) | .

Original: These two conditions are clearly directly analogous to in the m > 0 case, where b now acts as a withcondition (13) being b > 4 | m | .
Revised: These two conditions are clearly directly analogous to in the m > 0 case, where b now acts as a withcondition (13) being b > 4 | m | .

Original: Therefore the proof is complete.
Revised: Therefore the proof is complete.

Original: A Sparse Combo Net Details
Revised: A Sparse Combo Net Details

Original: Here we provide comprehensive information on the methodology and results for Sparse Combo Net,including some supplementary experimental results.
Revised: Here we provide comprehensive information on the methodology and results for Sparse Combo Net,including some supplementary experimental results.

Original: See the Appendix table of contents for a guideto this section.
Revised: See the Appendix table of contents for a guideto this section.

Original: A4.1 Extended MethodsA4.1.1 Initialization and Training
Revised: A4.1 Extended Methods

Revised: A4.1.1 Initialization and Training

Original: As described in the main text, the nonlinear RNN weights for Sparse Combo Net were randomlygenerated based on given sparsity and entry magnitude settings, and then conﬁrmed to meet the
Revised: As described in the main text, the nonlinear RNN weights for Sparse Combo Net were randomlygenerated based on given sparsity and entry magnitude settings, and then conﬁrmed to meet the

Original: Theorem 1 condition (or discarded if not).
Revised: Theorem 1 condition (or discarded if not).

Original: For a sparsity level of x and a magnitude limit of y , eachsubnetwork W was generated by drawing uniformly from between − y and y with x % density usingscipy.sparse.random, and then zeroing out the diagonal entries.
Revised: For a sparsity level of x and a magnitude limit of y , eachsubnetwork W was generated by drawing uniformly from between − y and y with x % density usingscipy.sparse.random, and then zeroing out the diagonal entries.

Original: For various potential x and y settings,we quantiﬁed both the likelihood that a generated W would satisfy Theorem 1, and the resultingnetwork performance.
Revised: For various potential x and y settings,we quantiﬁed both the likelihood that a generated W would satisfy Theorem 1, and the resultingnetwork performance.

Original: Of course this is also dependent on subnetwork size, as larger subnetworksenable greater sparsity.
Revised: Of course this is also dependent on subnetwork size, as larger subnetworksenable greater sparsity.

Original: The information we have obtained so far is documented in Section A4.2, inparticular A4.2.2.
Revised: The information we have obtained so far is documented in Section A4.2, inparticular A4.2.2.

Original: In training the linear connections between the described nonlinear RNN subnetworks, we constrainedthe matrix B in (3) to reﬂect underlying modularity assumptions.
Revised: In training the linear connections between the described nonlinear RNN subnetworks, we constrainedthe matrix B in (3) to reﬂect underlying modularity assumptions.

Original: In particular, we only train theoff-diagonal blocks of B and mask the diagonal blocks.
Revised: In particular, we only train theoff-diagonal blocks of B and mask the diagonal blocks.

Original: We do this to maintain the interpretation of
Revised: We do this to maintain the interpretation of

Original: L as the matrix containing the connection weights between different modules, as diagonal blockswould correspond to self-connections.
Revised: L as the matrix containing the connection weights between different modules, as diagonal blockswould correspond to self-connections.

Original: Furthermore, we only train the lower-triangular blocks of Bwhile masking the others, to increase training speed.
Revised: Furthermore, we only train the lower-triangular blocks of Bwhile masking the others, to increase training speed.

Original: To obtain the subnetwork RNN metrics necessary for training these linear connections,scipy.integrate.quad was used with default settings to solve for M in the equation − I =
Revised: To obtain the subnetwork RNN metrics necessary for training these linear connections,scipy.integrate.quad was used with default settings to solve for M in the equation − I =

Original: MW +
Revised: MW +

Original: W T M , as described in the main text.
Revised: W T M , as described in the main text.

Original: This was done by integrating e W T t Q e W t dt from 0 to ∞ .
Revised: This was done by integrating e W T t Q e W t dt from 0 to ∞ .

Original: Forefﬁciency reasons, and due to the guaranteed existence of a diagonal metric in the case of Theorem1, integration was only performed to solve for the diagonal elements of M .
Revised: Forefﬁciency reasons, and due to the guaranteed existence of a diagonal metric in the case of Theorem1, integration was only performed to solve for the diagonal elements of M .

Original: Therefore a check wasadded prior to training to conﬁrm that the initialized network indeed satisﬁed Theorem 1 with metric
Revised: Therefore a check wasadded prior to training to conﬁrm that the initialized network indeed satisﬁed Theorem 1 with metric

Original: M .
Revised: M .

Original: However, it was never triggered by our initialization method.
Revised: However, it was never triggered by our initialization method.

Original: Initial training hyperparameter tuning was done primarily with 10 × 16 combination networks onthe permuted seqMNIST task, starting with settings based on existing literature on this task, andverifying promising settings using a 15 × 16 network (Table S2).
Revised: Initial training hyperparameter tuning was done primarily with 10 × 16 combination networks onthe permuted seqMNIST task, starting with settings based on existing literature on this task, andverifying promising settings using a 15 × 16 network (Table S2).

Original: Initialization settings were held thesame throughout, as was later done for the size comparison trials (described in Section A4.2.1).
Revised: Initialization settings were held thesame throughout, as was later done for the size comparison trials (described in Section A4.2.1).

Original: Once hyperparameters were decided upon, the trials reported on in the main text began.
Revised: Once hyperparameters were decided upon, the trials reported on in the main text began.

Original: Most ofthese experiments were also done on permuted seqMNIST, where we characterized performance ofnetworks with different sizes and sparsity levels/entry magnitudes.
Revised: Most ofthese experiments were also done on permuted seqMNIST, where we characterized performance ofnetworks with different sizes and sparsity levels/entry magnitudes.

Original: When we moved to the sequential
Revised: When we moved to the sequential

Original: CIFAR10 task, we began by simply training with the same best settings that were found from theseexperiments.
Revised: CIFAR10 task, we began by simply training with the same best settings that were found from theseexperiments.

Original: The results of all attempted trials are reported in Section A4.4.
Revised: The results of all attempted trials are reported in Section A4.4.

Original: Unless speciﬁed otherwise, all networks reported on in the main text were trained for 150 epochs,using an Adam optimizer with initial learning rate 1e-3 and weight decay 1e-5.
Revised: Unless speciﬁed otherwise, all networks reported on in the main text were trained for 150 epochs,using an Adam optimizer with initial learning rate 1e-3 and weight decay 1e-5.

Original: The learning rate wascut to 1e-4 after 90 epochs and to 1e-5 after 140.
Revised: The learning rate wascut to 1e-4 after 90 epochs and to 1e-5 after 140.

Original: After identifying the most promising settings, weran repetitions trials on the best networks for 200 epochs with learning rate cuts after epochs 140 and190.
Revised: After identifying the most promising settings, weran repetitions trials on the best networks for 200 epochs with learning rate cuts after epochs 140 and190.

Original: A4.1.2 Code and Datasets
Revised: A4.1.2 Code and Datasets

Original: All Sparse Combo Nets described in the main text were trained using a single GPU on Google Colab.
Revised: All Sparse Combo Nets described in the main text were trained using a single GPU on Google Colab.

Original: Code to replicate all experiments can be found here: https://colab.research.google.com/drive/1JCT5OMgaMVK_Xh8BDFNRrEsyF0Ojvg10?usp=sharing
Revised: Code to replicate all experiments can be found here: https://colab.research.google.com/drive/1JCT5OMgaMVK_Xh8BDFNRrEsyF0Ojvg10?usp=sharing

Original: Runtime for the best performing architecture settings on the sequential CIFAR10 task was ∼ 24hours.
Revised: Runtime for the best performing architecture settings on the sequential CIFAR10 task was ∼ 24hours.

Original: A Colab Pro+ account was used to limit timeouts and prioritize GPU access.
Revised: A Colab Pro+ account was used to limit timeouts and prioritize GPU access.

Original: The datasets we used for our tasks were MNIST and CIFAR10, downloaded via PyTorch.
Revised: The datasets we used for our tasks were MNIST and CIFAR10, downloaded via PyTorch.

Original: MNISTis a handwritten digits classiﬁcation task, consisting of 60,000 training images and 10,000 testingimages (each 28x28 and grayscale).
Revised: MNISTis a handwritten digits classiﬁcation task, consisting of 60,000 training images and 10,000 testingimages (each 28x28 and grayscale).

Original: It is made available under the terms of the Creative Commons
Revised: It is made available under the terms of the Creative Commons

Original: Attribution-Share Alike 3.0 license.
Revised: Attribution-Share Alike 3.0 license.

Original: CIFAR10 is a dataset of 32x32 color images, split evenly amongthe following 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
Revised: CIFAR10 is a dataset of 32x32 color images, split evenly amongthe following 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

Original: Italso contains 60,000/10,000 training/test images, and is distributed under the MIT License.
Revised: Italso contains 60,000/10,000 training/test images, and is distributed under the MIT License.

Original: As mentioned in the main text, we presented these images to our networks pixel by pixel.
Revised: As mentioned in the main text, we presented these images to our networks pixel by pixel.

Original: In the caseof MNIST, we used an additional modiﬁcation to the dataset by permuting the pixels in a randomlydetermined (but ﬁxed across the dataset) way.
Revised: In the caseof MNIST, we used an additional modiﬁcation to the dataset by permuting the pixels in a randomlydetermined (but ﬁxed across the dataset) way.

Original: The use of these datasets is included in the above linkto our code.
Revised: The use of these datasets is included in the above linkto our code.

Original: Extended results information begins on the next page.
Revised: Extended results information begins on the next page.

Original: A4.2 Extended Results
Revised: A4.2 Extended Results

Original: In this subsection, we describe additional results we did not get to in the main text, related toscalability, modularity, sparsity, and repeatability.
Revised: In this subsection, we describe additional results we did not get to in the main text, related toscalability, modularity, sparsity, and repeatability.

Original: We also add some further discussion of theseresults, along with more detailed information on the respective experimental set ups.
Revised: We also add some further discussion of theseresults, along with more detailed information on the respective experimental set ups.

Original:  A4.2.1 Network Size and Modularity Comparison
Revised:  A4.2.1 Network Size and Modularity Comparison

Original: For the Sparse Combo Net speciﬁcally we had additional experiments on architecture size and unitdistribution besides what was depicted in the main text.
Revised: For the Sparse Combo Net speciﬁcally we had additional experiments on architecture size and unitdistribution besides what was depicted in the main text.

Original: The results of these supplemental experimentsboth replicated the observed effect in Figure 4 of network size instead using subnetworks with 16units each as well as higher density (Figure S1A), and evaluated how task performance varies withmodularity of a network ﬁxed to have 352 total units (Figure S1B).
Revised: The results of these supplemental experimentsboth replicated the observed effect in Figure 4 of network size instead using subnetworks with 16units each as well as higher density (Figure S1A), and evaluated how task performance varies withmodularity of a network ﬁxed to have 352 total units (Figure S1B).

Original: In the modularity experiment weobserved an inverse U shape, with poor performance of a 1 × 352 net and an 88 × 4 net, and bestperformance from a 44 × 8 net.
Revised: In the modularity experiment weobserved an inverse U shape, with poor performance of a 1 × 352 net and an 88 × 4 net, and bestperformance from a 44 × 8 net.

Original: Note that this experiment compared similar sparsity levels across thedifferent subnetwork sizes.
Revised: Note that this experiment compared similar sparsity levels across thedifferent subnetwork sizes.

Original: In practice we can achieve better performance with larger subnetworks byleveraging sparsity in a way not possible in smaller subnetworks.
Revised: In practice we can achieve better performance with larger subnetworks byleveraging sparsity in a way not possible in smaller subnetworks.

Original: These additional experiments arenow described in more detail below.
Revised: These additional experiments arenow described in more detail below.

Original: For the initial round of size comparison trials using subnetworks of 16 units each (Figure S1A), thenonlinear RNN weights were set by drawing uniformly from between − 0 .
Revised: For the initial round of size comparison trials using subnetworks of 16 units each (Figure S1A), thenonlinear RNN weights were set by drawing uniformly from between − 0 .

Original: 4 and 0 .
Revised: 4 and 0 .

Original: 4 with 40% densityusing scipy.sparse.random, and then zeroing out the diagonal entries.
Revised: 4 with 40% densityusing scipy.sparse.random, and then zeroing out the diagonal entries.

Original: These settings were chosenbecause they resulted in ∼ 1% of 16 by 16 weight matrices meeting the Theorem 1 condition.
Revised: These settings were chosenbecause they resulted in ∼ 1% of 16 by 16 weight matrices meeting the Theorem 1 condition.

Original: Duringinitialization only the matrices meeting this condition were kept, ﬁnishing when the desired numberof component RNNs had been set - producing a block diagonal W like pictured in Figure 3A. Thissame initialization process was used throughout our experiments.
Revised: Duringinitialization only the matrices meeting this condition were kept, ﬁnishing when the desired numberof component RNNs had been set - producing a block diagonal W like pictured in Figure 3A. Thissame initialization process was used throughout our experiments.

Original: In later experiments we vary thedensity and magnitude settings.
Revised: In later experiments we vary thedensity and magnitude settings.

Original: For the size experiments, we held static the number of units and initialization settings for eachcomponent RNN, and tested the effect of changing the number of components in the combinationnetwork.
Revised: For the size experiments, we held static the number of units and initialization settings for eachcomponent RNN, and tested the effect of changing the number of components in the combinationnetwork.

Original: 1, 3, 5, 10, 15, 20, 22, 25, and 30 components were tested in this experiment (Figure
Revised: 1, 3, 5, 10, 15, 20, 22, 25, and 30 components were tested in this experiment (Figure

Original: S1A).
Revised: S1A).

Original: Increasing the number of components initially lead to great improvements in test accuracy,but had diminishing returns - test accuracy consistently hit ∼ 93% with a large enough numberof subnetworks, but neither loss nor accuracy showed meaningful improvement past the 22 × 16network.
Revised: Increasing the number of components initially lead to great improvements in test accuracy,but had diminishing returns - test accuracy consistently hit ∼ 93% with a large enough numberof subnetworks, but neither loss nor accuracy showed meaningful improvement past the 22 × 16network.

Original: Interestingly, early training loss and accuracy became substantially worse once the numberof components increased past a certain point, falling from 70% to 43% epoch 1 test accuracy betweenthe 22 × 16 and 30 × 16 networks.
Revised: Interestingly, early training loss and accuracy became substantially worse once the numberof components increased past a certain point, falling from 70% to 43% epoch 1 test accuracy betweenthe 22 × 16 and 30 × 16 networks.

Original: The complete set of results can be found in Table S3.
Revised: The complete set of results can be found in Table S3.

Original: Note that the size experiment described in the main text (Figure 4A) was a repetition of this originalexperiment, but now using 32 unit subnetworks with the best performing sparsity settings.
Revised: Note that the size experiment described in the main text (Figure 4A) was a repetition of this originalexperiment, but now using 32 unit subnetworks with the best performing sparsity settings.

Original: The resultsfor the repetition can be found in Table S5.
Revised: The resultsfor the repetition can be found in Table S5.

Original: To better understand how the modularity of the combination networks affects performance, the nextexperiment held the number of total units constant at 352, selected due to the prior success of the22 × 16 network, and tested different allocations of these units amongst component RNNs.
Revised: To better understand how the modularity of the combination networks affects performance, the nextexperiment held the number of total units constant at 352, selected due to the prior success of the22 × 16 network, and tested different allocations of these units amongst component RNNs.

Original: Thus1 × 352 , 11 × 32 , 44 × 8 , and 88 × 4 networks were trained to compare against the 22 × 16 (Figure
Revised: Thus1 × 352 , 11 × 32 , 44 × 8 , and 88 × 4 networks were trained to compare against the 22 × 16 (Figure

Original: S1B).
Revised: S1B).

Original: Increasing the modularity improved performance to a point, with the 44 × 8 network resultingin ﬁnal test accuracy of 94 .
Revised: Increasing the modularity improved performance to a point, with the 44 × 8 network resultingin ﬁnal test accuracy of 94 .

Original: 44% , while conversely the 11 × 32 resulted in decreased test accuracy.
Revised: 44% , while conversely the 11 × 32 resulted in decreased test accuracy.

Original: However, the 88 × 4 network was unable to learn, and a 352 × 1 network would theoretically just bea scaled linear anti-symmetric network.
Revised: However, the 88 × 4 network was unable to learn, and a 352 × 1 network would theoretically just bea scaled linear anti-symmetric network.

Original: Because larger networks require different sparsity settings to meet the Theorem 1 condition, thesewere not held constant between trials in the modularity comparison experiment (Figure S1B), butrather selected in the same way between trials - looking for settings that keep density and scalarbalanced and result in ∼ 1% of the matrices meeting the condition.
Revised: Because larger networks require different sparsity settings to meet the Theorem 1 condition, thesewere not held constant between trials in the modularity comparison experiment (Figure S1B), butrather selected in the same way between trials - looking for settings that keep density and scalarbalanced and result in ∼ 1% of the matrices meeting the condition.

Original: The scalar was applied aftersampling non-zero entries from a uniform distribution between -1 and 1.
Revised: The scalar was applied aftersampling non-zero entries from a uniform distribution between -1 and 1.

Original: The resulting settings were7.5% density and 0.077 scalar for 352 unit component RNN, 26.5% density and 0.27 scalar for 32unit component RNN, 60% density and 0.7 scalar for 8 unit component RNN, and 100% density and1.0 scalar for 4 unit component RNN.
Revised: The resulting settings were7.5% density and 0.077 scalar for 352 unit component RNN, 26.5% density and 0.27 scalar for 32unit component RNN, 60% density and 0.7 scalar for 8 unit component RNN, and 100% density and1.0 scalar for 4 unit component RNN.

Original: The complete set of results for the modularity experiment canbe found in Table S4.
Revised: The complete set of results for the modularity experiment canbe found in Table S4.

Original: A4.2.2 Sparsity Settings Comparison
Revised: A4.2.2 Sparsity Settings Comparison

Original: Density and scalar settings for the component nonlinear RNNs were initially chosen for each networksize using the percentage of random networks that met the Theorem 1 condition.
Revised: Density and scalar settings for the component nonlinear RNNs were initially chosen for each networksize using the percentage of random networks that met the Theorem 1 condition.

Original: For scalar s , acomponent network would have non-zero entries sampled uniformly between − s and s .
Revised: For scalar s , acomponent network would have non-zero entries sampled uniformly between − s and s .

Original: When we began experimenting with sparsity in the initialization (after seeing the large performancedifference in Figure 5), we split the previously described scalar setting into two different scalarsone applied before a random matrix was checked against the Theorem 1 condition, and one appliedafter a matrix was selected.
Revised: When we began experimenting with sparsity in the initialization (after seeing the large performancedifference in Figure 5), we split the previously described scalar setting into two different scalarsone applied before a random matrix was checked against the Theorem 1 condition, and one appliedafter a matrix was selected.

Original: Of course the latter must be ≤ 1 to guarantee stability is preserved.
Revised: Of course the latter must be ≤ 1 to guarantee stability is preserved.

Original: Thescalar was separated out after we noticed that at 5% density, random 32 by 32 weight matrices metthe condition roughly 1% of the time whether the scalar was 10 or 100000 - ∼ 85% of sampledmatrices using scalar 10 would continue to meet the condition even if multiplied by a factor of 10000 .
Revised: Thescalar was separated out after we noticed that at 5% density, random 32 by 32 weight matrices metthe condition roughly 1% of the time whether the scalar was 10 or 100000 - ∼ 85% of sampledmatrices using scalar 10 would continue to meet the condition even if multiplied by a factor of 10000 .

Original: Therefore we wanted a mechanism that could bias selection towards matrices that are stable due totheir sparsity and not due to magnitude constraints, while still keeping the elements to a reasonablesize for training purposes.
Revised: Therefore we wanted a mechanism that could bias selection towards matrices that are stable due totheir sparsity and not due to magnitude constraints, while still keeping the elements to a reasonablesize for training purposes.

Original: Ultimately, both sparsity and magnitude had a clear effect on performance (Figure S2).
Revised: Ultimately, both sparsity and magnitude had a clear effect on performance (Figure S2).

Original: Increasesin both had a positive correlation with accuracy and loss through most parameters tested.
Revised: Increasesin both had a positive correlation with accuracy and loss through most parameters tested.

Original: Best testaccuracy overall was 96 .
Revised: Best testaccuracy overall was 96 .

Original: 79% , which was obtained by both a 16 × 32 network with 5% density andentries between -5 and 5, and a 16 × 32 network with 3 . 3% density and entries between -6 and 6.
Revised: 79% , which was obtained by both a 16 × 32 network with 5% density andentries between -5 and 5, and a 16 × 32 network with 3 . 3% density and entries between -6 and 6.

Original: The latter also achieved the best epoch 1 test accuracy observed of 86 .
Revised: The latter also achieved the best epoch 1 test accuracy observed of 86 .

Original: Thus we chose to gowith these settings for our extended training repetitions on permuted seqMNIST.
Revised: Thus we chose to gowith these settings for our extended training repetitions on permuted seqMNIST.

Original: It is also worth noting that upon investigation of the subnetwork weight matrices across these trials,the sparser networks had substantially lower maximum eigenvalue of | W | , suggesting that strongerstability can actually correlate with improved performance on sequential tasks.
Revised: It is also worth noting that upon investigation of the subnetwork weight matrices across these trials,the sparser networks had substantially lower maximum eigenvalue of | W | , suggesting that strongerstability can actually correlate with improved performance on sequential tasks.

Original: This could be due to amechanism such as that described in [Radhakrishnana et al., 2020].
Revised: This could be due to amechanism such as that described in [Radhakrishnana et al., 2020].

Original: Results from the initial trials of different sparsity levels across different network sizes can be foundin Table S6.
Revised: Results from the initial trials of different sparsity levels across different network sizes can be foundin Table S6.

Original: Results from the more thorough testing of different sparsity levels and magnitudes in a16 × 32 network can be found in Table S7.
Revised: Results from the more thorough testing of different sparsity levels and magnitudes in a16 × 32 network can be found in Table S7.

Original: A4.2.3 Repeatability Tests
Revised: A4.2.3 Repeatability Tests

Original:  Here we give additional details on the repeatability tests described in the main text, as well asintroducing some additional results using other hyperparameters (done after the initial SOTA-settingexperiments) on sequential CIFAR10.
Revised: [Figure] Here we give additional details on the repeatability tests described in the main text, as well asintroducing some additional results using other hyperparameters (done after the initial SOTA-settingexperiments) on sequential CIFAR10.

Original: To further improve performance once network settings were explored on permuted seqMNIST, anextended training run was tested on the best performing option.
Revised: To further improve performance once network settings were explored on permuted seqMNIST, anextended training run was tested on the best performing option.

Original: Settings were kept the same as aboveusing a 3 . 3% density 16 ×
Revised: Settings were kept the same as aboveusing a 3 . 3% density 16 ×

Original: 32 network, except training now ran for over 200 epochs, with just a singlelearning rate cut occurring after epoch 200 (exact number of epochs varied based on runtime limit).
Revised: 32 network, except training now ran for over 200 epochs, with just a singlelearning rate cut occurring after epoch 200 (exact number of epochs varied based on runtime limit).

Original: This experiment was repeated four times and resulted in 96 .
Revised: This experiment was repeated four times and resulted in 96 .

Original: 94% best test accuracy, as described inthe main text (Figure S3).
Revised: 94% best test accuracy, as described inthe main text (Figure S3).

Original: Table S8 reports additional details on these trials.
Revised: Table S8 reports additional details on these trials.

Original: As mentioned, we also characterized Sparse Combo Net performance on the more challenging
Revised: As mentioned, we also characterized Sparse Combo Net performance on the more challenging

Original: CIFAR10 sequential image classiﬁcation task, across a greater number of trials.
Revised: CIFAR10 sequential image classiﬁcation task, across a greater number of trials.

Original: Ten trials wererun for 200 epochs with learning rate scaled after epochs 140 and 190.
Revised: Ten trials wererun for 200 epochs with learning rate scaled after epochs 140 and 190.

Original: All other settings were thesame as for the permuted seqMNIST repeatability trials.
Revised: All other settings were thesame as for the permuted seqMNIST repeatability trials.

Original: As reported in the main text, the mean testaccuracy observed was 64.72%, with variance 0.406, and range 63.73%-65.72%.
Revised: As reported in the main text, the mean testaccuracy observed was 64.72%, with variance 0.406, and range 63.73%-65.72%.

Original: Training loss andtest accuracy over the course of learning are plotted for all ten trials in Figure S4, and exact numbersfor each trial are provided in Table S10.
Revised: Training loss andtest accuracy over the course of learning are plotted for all ten trials in Figure S4, and exact numbersfor each trial are provided in Table S10.

Original: As we encountered difﬁculties with Colab GPU assignment when we started working on theserepetitions, we also trained nine networks over a smaller number of epochs (Figure S5).
Revised: As we encountered difﬁculties with Colab GPU assignment when we started working on theserepetitions, we also trained nine networks over a smaller number of epochs (Figure S5).

Original: Thesenetworks were trained using the 150 epoch paradigm previously described, although only four of thenine completed training within the 24 hour runtime limit.
Revised: Thesenetworks were trained using the 150 epoch paradigm previously described, although only four of thenine completed training within the 24 hour runtime limit.

Original: Complete results for these trials can be found in Table S11.
Revised: Complete results for these trials can be found in Table S11.

Original: Mean performance among the shorter training trials was 62.82% test accuracywith variance 0.95.
Revised: Mean performance among the shorter training trials was 62.82% test accuracywith variance 0.95.

Original: Prior to beginning the repeatability experiments on seqCIFAR we explored alternative hyperparameters on this task (Table S9).
Revised: Prior to beginning the repeatability experiments on seqCIFAR we explored alternative hyperparameters on this task (Table S9).

Original: While we ultimately ended up using the same hyperparameters asfor permuted seqMNIST, these results further support the robustness of our architecture.
Revised: While we ultimately ended up using the same hyperparameters asfor permuted seqMNIST, these results further support the robustness of our architecture.

Original: To complete our benchmarking table we also ran a single 150 epoch trial of our best network settingson the sequential MNIST task.
Revised: To complete our benchmarking table we also ran a single 150 epoch trial of our best network settingson the sequential MNIST task.

Original: Test accuracy over the course of training for this trial is depicted in
Revised: Test accuracy over the course of training for this trial is depicted in

Original: Figure S6.
Revised: Figure S6.

Original:  training (done to complete more trials)
Revised: mance repeated trials shorter training (done to complete more trials).

Original: Nine different 16 × 32 networks with 3.3% density and entries between -6 and 6 were set up to train for 150 epochs, with learning rate divided by 10 after epochs 90 and 140.
Revised: Nine different 16 × 32 networks with 3.3% density and entries between -6 and 6 were set up to train for 150 epochs, with learning rate divided by 10 after epochs 90 and 140.

Original: Most of these networks hit runtime limit before completing, however they all got through at leastepochs and all had test accuracy exceed 61% the course of training that completed training are  lines  were 
Revised: Most of these networks hit runtime limit before completing, however they all got through at leastepochs and all had test accuracy exceed 61%. This accuracy for each of the networks over the course of training that completed training are plotted as solid lines those that were cut short dashed

Original: A4.2.4 Scalability Pilot: Feedback Sparsity Tests
Revised: A4.2.4 Scalability Pilot: Feedback Sparsity Tests

Original: Not only did the 16 × 32 Sparse Combo Net with 3 . 3% density achieve test accuracy sequential
Revised: Not only did the 16 × 32 Sparse Combo Net with 3 . 3% density achieve test accuracy sequential

Original: CIFAR10 that is the highest to date for a provably stable RNN, it was higher than the 1 millionparameter CKConv network, which set a recent SOTA for permuted seqMNIST accuracy (Table 1).
Revised: CIFAR10 that is the highest to date for a provably stable RNN, it was higher than the 1 millionparameter CKConv network, which set a recent SOTA for permuted seqMNIST accuracy (Table 1).

Original: Our network has 130 , 000 trainable parameters by comparison.
Revised: Our network has 130 , 000 trainable parameters by comparison.

Original: Thus we achieve very impressiveresults given the characteristics of our architecture.
Revised: Thus we achieve very impressiveresults given the characteristics of our architecture.

Original: However, the results of the network size experiments do raise some concern about the scalability ofthe approach.
Revised: However, the results of the network size experiments do raise some concern about the scalability ofthe approach.

Original: Here we provide pilot results suggesting that this issue can likely be addressed via theintroduction of sparsity in the linear inter-subnetwork connectivity matrix L .
Revised: Here we provide pilot results suggesting that this issue can likely be addressed via theintroduction of sparsity in the linear inter-subnetwork connectivity matrix L .

Original: While all results in themain text use all-to-all negative feedback, we have early results suggesting that in larger networks,fewer negative feedback connections may perform better, thus preventing performance saturationwith scaling.
Revised: While all results in themain text use all-to-all negative feedback, we have early results suggesting that in larger networks,fewer negative feedback connections may perform better, thus preventing performance saturationwith scaling.

Original: Below are the results of testing of this idea in a 24 × 32 Sparse Combo Net on the sequential CIFAR10task.
Revised: Below are the results of testing of this idea in a 24 × 32 Sparse Combo Net on the sequential CIFAR10task.

Original: We varied the number of feedback connections that were ﬁxed at 0 while all other settingsremained static (Table S1).
Revised: We varied the number of feedback connections that were ﬁxed at 0 while all other settingsremained static (Table S1).

Original: The resulting performance took an inverse U shape, where the networkwith only 50% of possible feedback connections non-zero had the best test accuracy of 65.14%,achieved in just 124 epochs of training.
Revised: The resulting performance took an inverse U shape, where the networkwith only 50% of possible feedback connections non-zero had the best test accuracy of 65.14%,achieved in just 124 epochs of training.

Original: [Table]
Revised: [Table]

Original: Decreasing Feedback Density is a promising path towards further improving performance as thesize of scaled
Revised: Decreasing Feedback Density is a promising path towards further improving performance as the scaled

Original: A4.3 Architecture Interpretability
Revised: A4.3 Architecture Interpretability

Original: In this subsection, we give additional information on trainable parameters and properties of thenetwork weights, to assist in interpreting the Sparse Combo Net architecture and its training process.
Revised: In this subsection, we give additional information on trainable parameters and properties of thenetwork weights, to assist in interpreting the Sparse Combo Net architecture and its training process.

Original: A4.3.1 Number of Parameters
Revised: A4.3.1 Number of Parameters

Original: To report on the number of trainable parameters, we used the following formula:
Revised: To report on the number of trainable parameters, we used the following formula:

Original: [Equation]
Revised: [Equation]

Original: Where n is the total number of units in the M × C combination network, o is the total number ofoutput nodes for the task, and i is the total number of input nodes for the task.
Revised: Where n is the total number of units in the M × C combination network, o is the total number ofoutput nodes for the task, and i is the total number of input nodes for the task.

Original: Thus for the 16 × 32networks highlighted here, we have 129034 trainable parameters for the MNIST tasks, and 130058trainable parameters for sequential CIFAR10.
Revised: Thus for the 16 × 32networks highlighted here, we have 129034 trainable parameters for the MNIST tasks, and 130058trainable parameters for sequential CIFAR10.

Original: Note that the naive estimate for the number of trainable parameters would be n 

Revised: Note that the naive estimate for the number of trainable parameters would be n

Original: + i ∗ n + n ∗ o + n + o ,corresponding to the number of weights in L , the number of weights in the feedforward linear inputlayer, the number of weights in the feedforward linear output layer, and the bias terms for the inputand output layers, respectively.
Revised: + i ∗ n + n ∗ o + n + o ,corresponding to the number of weights in L , the number of weights in the feedforward linear inputlayer, the number of weights in the feedforward linear output layer, and the bias terms for the inputand output layers, respectively.

Original: However, because of the combination property constraints on L , onlythe lower triangular portion of a block off-diagonal matrix is actually trained, and L is then deﬁnedin terms of this matrix and the metric M .
Revised: However, because of the combination property constraints on L , onlythe lower triangular portion of a block off-diagonal matrix is actually trained, and L is then deﬁnedin terms of this matrix and the metric M .

Original: Thus we subtract M ∗ C 2 to remove the block diagonalportions corresponding to nonlinear RNN components, and then divide by 2 to obtain only the lowerhalf.
Revised: Thus we subtract M ∗ C 2 to remove the block diagonalportions corresponding to nonlinear RNN components, and then divide by 2 to obtain only the lowerhalf.

Original: A4.3.2 Inspecting Trained Network Weights
Revised: A4.3.2 Inspecting Trained Network Weights

Original: After training was completed, we inspected the state of all networks described in the main text,pulling both the nonlinear ( W ) and linear ( L ) weight matrices from both initialization time and theﬁnal model.
Revised: After training was completed, we inspected the state of all networks described in the main text,pulling both the nonlinear ( W ) and linear ( L ) weight matrices from both initialization time and theﬁnal model.

Original: For W , we conﬁrmed it did not change over training, and inspected the max real part ofthe eigenvalues of | W | in accordance with Theorem 1.
Revised: For W , we conﬁrmed it did not change over training, and inspected the max real part ofthe eigenvalues of | W | in accordance with Theorem 1.

Original: The densest tested matrices tended to haveλ max ( | W | ) > 0 .
Revised: The densest tested matrices tended to haveλ max ( | W | ) > 0 .

Original: 9 , while the sparsest ones tended to have λ max ( | W | ) < 0 .
Revised: 9 , while the sparsest ones tended to have λ max ( | W | ) < 0 .

Original: For L , we checked themaximum element and the maximum singular value before and after training.
Revised: For L , we checked themaximum element and the maximum singular value before and after training.

Original: In general, both wentup over the course of training, but by a modest amount.
Revised: In general, both wentup over the course of training, but by a modest amount.

Original: A4.4 Tables of Results by Trial (Supplementary Documentation)
Revised: A4.4 Tables of Results by Trial (Supplementary Documentation)

Original: Table S2 shows all trials run on permuted sequential MNIST before beginning the more systematicexperiments reported on in the main text.
Revised: Table S2 shows all trials run on permuted sequential MNIST before beginning the more systematicexperiments reported on in the main text.

Original: Notably, our networks did not require an extensivehyperparameter tuning process.
Revised: Notably, our networks did not require an extensivehyperparameter tuning process.

Original: Tables S3 and S4 report additional details on the initial size and modularity experiments (Figure S1).
Revised: Tables S3 and S4 report additional details on the initial size and modularity experiments (Figure S1).

Original: Table S5 reports the results of the repeated experiment on Sparse Combo Net size, this time using 32unit component subnetworks with best sparsity settings (Figure 4A).
Revised: Table S5 reports the results of the repeated experiment on Sparse Combo Net size, this time using 32unit component subnetworks with best sparsity settings (Figure 4A).

Original: Tables S6 and S7 report results from all trials related to our sparsity experiments (Figures 5 and S2).
Revised: Tables S6 and S7 report results from all trials related to our sparsity experiments (Figures 5 and S2).

Original: Table S8 provides further information on the four trials in the permuted seqMNIST repeatabilityexperiments (Figure S3).
Revised: Table S8 provides further information on the four trials in the permuted seqMNIST repeatabilityexperiments (Figure S3).

Original: Table S9 reports the results of all trials of different hyperparameters on the sequential CIFAR10 task,in chronological order.
Revised: Table S9 reports the results of all trials of different hyperparameters on the sequential CIFAR10 task,in chronological order.

Original: Ultimately the same settings as those used for permuted seqMNIST werechosen.
Revised: Ultimately the same settings as those used for permuted seqMNIST werechosen.

Original: Table S10 shows the results of all ten trials in the seqCIFAR10 repeatability experiments (Figure S4).
Revised: Table S10 shows the results of all ten trials in the seqCIFAR10 repeatability experiments (Figure S4).

Original: Table S11 shows results from nine additional seqCIFAR10 trials of shorter training duration (Figure
Revised: Table S11 shows results from nine additional seqCIFAR10 trials of shorter training duration (Figure

Original: S5), run to increase sample size while unable to access the higher quality Colab GPUs.
Revised: S5), run to increase sample size while unable to access the higher quality Colab GPUs.

Original: Finally, Table S1 reports the results of our pilot trial on introducing sparsity into the linear feedbackconnection matrix - as this table was presented in Section A4.2.4 however, we do not reproduce ithere.A
Revised: Finally, Table S1 reports the results of our pilot trial on introducing sparsity into the linear feedbackconnection matrix - as this table was presented in Section A4.2.4 however, we do not reproduce ithere.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: containing different numbers of component 16-unit RNNs.
Revised: containing different numbers of component 16-unit RNNs.

Original: Training hyperparameters and network initialization settings were kept the same across all trials, and all trials completed the full 150 epochs.
Revised: Training hyperparameters and network initialization settings were kept the same across all trials, and all trials completed the full 150 epochs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A4.4.2 seqCIFAR10 Trials
Revised: A4.4.2 seqCIFAR10 Trials

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The mean test accuracy achieved was 64.72% with Variance 0.406.
Revised: The mean test accuracy achieved was 64.72% with Variance 0.406.

Original: Most trials completed all 200 epochs, but two were cut short due
Revised: Most trials completed all 200 epochs, but two were cut short due

Original: [Table]
Revised: [Table]

Original: - qCIFAR10 task.
Revised: - qCIFAR10 task.

Original: For these trials we used the same 150 epoch training paradigm as previously, although only four of the networks were able to fully complete training.
Revised: For these trials we used the same 150 epoch training paradigm as previously, although only four of the networks were able to fully complete training.

Original: These trials were done to expand our sample size while access was limited to only slower GPUs.
Revised: These trials were done to expand our sample size while access was limited to only slower GPUs.

Original: The mean observed test accuracy among the shorter trials was 62.82%, with variance of 0.95.
Revised: The mean observed test accuracy among the shorter trials was 62.82%, with variance of 0.95.

Original: A SVD Combo Net DetailsA5.1 Parameterization Information
Revised: A SVD Combo Net DetailsA5.1 Parameterization Information

Original: For the SVD Combo Net, we ensured contraction by directly parameterizing each of the W i( i = 1 , 2 . . .
Revised: For the SVD Combo Net, we ensured contraction by directly parameterizing each of the W i( i = 1 , 2 . . .

Original: p ) as:
Revised: p ) as:

Original: [Equation]
Revised: [Equation]

Original: (14) where Φ i is diagonal and nonsingular, U i and V
Revised: (14) where Φ i is diagonal and nonsingular, U i and V

Original: i are orthogonal, and Σ i is diagonal with Σ ii ∈[0 , g − 1 ) .
Revised: i are orthogonal, and Σ i is diagonal with Σ ii ∈[0 , g − 1 ) .

Original: We ensure orthogonality of U i and V i during training by exploiting the fact that thematrix exponential of a skew-symmetric matrix is orthogonal, as was done in [Lezcano-Casado and
Revised: We ensure orthogonality of U i and V i during training by exploiting the fact that thematrix exponential of a skew-symmetric matrix is orthogonal, as was done in [Lezcano-Casado and

Original: Martınez-Rubio, 2019].
Revised: Martınez-Rubio, 2019].

Original: The network constructed from these subnetworks using (2) is contracting inmetric ˜ M =
Revised: The network constructed from these subnetworks using (2) is contracting inmetric ˜ M =

Original: BlockDiag ( Φ 21 , . .
Revised: BlockDiag ( Φ 21 , . .

Original: Φ 2 p ) .
Revised: Φ 2 p ) .

Original: A5.2 Control Experiment
Revised: A5.2 Control Experiment

Original: To do the SVD Combo Network stability control experiment, we still constrained the weights ofthe subnetwork modules using the above parameterization, but we removed any constraint on theconnections between modules.
Revised: To do the SVD Combo Network stability control experiment, we still constrained the weights ofthe subnetwork modules using the above parameterization, but we removed any constraint on theconnections between modules.

Original: Thus each individual module was still guaranteed to be contracting,but the overall system had no such guarantee in this control trial.
Revised: Thus each individual module was still guaranteed to be contracting,but the overall system had no such guarantee in this control trial.

Original: The experiment was run using a24 × 32 combination network and the permuted sequential MNIST task.
Revised: The experiment was run using a24 × 32 combination network and the permuted sequential MNIST task.

Original: In contrast to the primary control experiment run on Sparse Combo Net, for the SVD Combo Netwe saw only a very slight performance decrease when the between-module connections were nolonger constrained for the overall stability certiﬁcate.
Revised: In contrast to the primary control experiment run on Sparse Combo Net, for the SVD Combo Netwe saw only a very slight performance decrease when the between-module connections were nolonger constrained for the overall stability certiﬁcate.

Original: Test accuracy in the control run was 94 .
Revised: Test accuracy in the control run was 94 .

Original: 56% ,as compared to the 94 .
Revised: 56% ,as compared to the 94 .

Original: 9% observed with the standard SVD Combo Net.
Revised: 9% observed with the standard SVD Combo Net.

Original: This disparity in performance decrease makes sense when considering that the hidden-to-hiddenweights of SVD Combo Network are trainable, while those of the Sparse Combo Network are not.
Revised: This disparity in performance decrease makes sense when considering that the hidden-to-hiddenweights of SVD Combo Network are trainable, while those of the Sparse Combo Network are not.

Original: Whatever instabilities are introduced by the lack of constraint on the inter-subnetwork connectionslikely cannot be adequately compensated for in the Sparse Combo Network.
Revised: Whatever instabilities are introduced by the lack of constraint on the inter-subnetwork connections [Figure] likely cannot be adequately compensated for in the Sparse Combo Network.

Original: [Figure]
Revised: [Figure]

Original: A5.3 ModelCode
Revised: .3 Model

Original: [Figure]
Revised: [Figure]