Original: I DENTIFYING NONLINEAR DYNAMICAL SYSTEMS WITH MULTIPLE TIME SCALES AND LONG - RANGE DEPENDENCIESAnonymous authors
Revised: I DENTIFYING NONLINEAR DYNAMICAL SYSTEMS WITH MULTIPLE TIME SCALES AND LONG - RANGE DEPENDENCIES

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A main theoretical interest in biology and physics is to identify the nonlinear dynamical system (DS) that generated observed time series.
Revised: A main theoretical interest in biology and physics is to identify the nonlinear dynamical system (DS) that generated observed time series.

Original: Recurrent Neural Networks (RNNs) are, in principle, powerful enough to approximate any underlying
Revised: Recurrent Neural Networks (RNNs) are, in principle, powerful enough to approximate any underlying

Original: DS, but in their vanilla form suffer from the exploding vs. vanishing gradients problem.
Revised: DS, but in their vanilla form suffer from the exploding vs. vanishing gradients problem.

Original: Previous attempts to alleviate this problem resulted either in more complicated, mathematically less tractable RNN architectures, or strongly limited the dynamical expressiveness of the RNN.
Revised: Previous attempts to alleviate this problem resulted either in more complicated, mathematically less tractable RNN architectures or strongly limited the dynamical expressiveness of the RNN.

Original: Here we address this issue by suggesting a simple regularization scheme for vanilla RNNs with ReLU activation which enables them to solve long-range dependency problems and express slow time scales, while retaining a simple mathematical structure which makes their DS properties partly analytically accessible.
Revised: Here we address this issue by suggesting a simple regularization scheme for vanilla RNNs with ReLU activation which enables them to solve long-range dependency problems and express slow time scales, while retaining a simple mathematical structure which makes their DS properties partly analytically accessible.

Original: We prove two theorems that establish a tight connection between the regularized RNN dynamics and their gradients, illustrate on
Revised: We prove two theorems that establish a tight connection between the regularized RNN dynamics and their gradients, illustrate on

Original: DS benchmarks that our regularization approach strongly eases the reconstruction of DS which harbor widely differing time scales, and show that our method is also en par with other long-range architectures like LSTMs on several tasks.
Revised: DS benchmarks that our regularization approach strongly eases the reconstruction of DS which harbor widely differing time scales and show that our method is also en par with other long-range architectures like LSTMs on several tasks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Theories in the natural sciences are often formulated in terms of sets of stochastic differential or difference equations, i.e. as stochastic dynamical systems (DS).
Revised: Theories in the natural sciences are often formulated in terms of sets of stochastic differential or difference equations, i.e. as stochastic dynamical systems (DS).

Original: Such systems exhibit a range of common phenomena, like (limit) cycles, chaotic attractors, or speciﬁc bifurcations, which are the subject of nonlinear dynamical systems theory (DST; Strogatz (2015); Ott (2002)).
Revised: Such systems exhibit a range of common phenomena, like (limit) cycles, chaotic attractors, or speciﬁc bifurcations, which are the subject of nonlinear dynamical systems theory (DST; Strogatz (2015); Ott (2002)).

Original: A long-standing desire is to retrieve the generating dynamical equations directly from observed time series data (Kantz & Schreiber, 2004), and thus to ‘automatize’ the laborious process of scientiﬁc theory building to some degree.
Revised: A long-standing desire is to retrieve the generating dynamical equations directly from observed time series data (Kantz & Schreiber, 2004), and thus to ‘automatize’ the laborious process of scientiﬁc theory building to some degree.

Original: A variety of machine and deep learning methodologies toward this goal have been introduced in recent years (Chen et al., 2017; Champion et al., 2019;
Revised: A variety of machine and deep learning methodologies toward this goal have been introduced in recent years (Chen et al., 2017; Champion et al., 2019;

Original: Ayed et al., 2019; Koppe et al., 2019; Hamilton et al., 2017; Razaghi & Paninski, 2019; Hernandez et al., 2020).
Revised: Ayed et al., 2019; Koppe et al., 2019; Hamilton et al., 2017; Razaghi & Paninski, 2019; Hernandez et al., 2020).

Original: Often these are based on sufﬁciently expressive series expansions for approximating the unknown system of generative equations, such as polynomial basis expansions (Brunton et al., 2016; Champion et al., 2019) or recurrent neural networks (RNNs) (Vlachas et al., 2018; Hernandez et al., 2020; Durstewitz, 2017;
Revised: Often these are based on sufﬁciently expressive series expansions for approximating the unknown system of generative equations, such as polynomial basis expansions (Brunton et al., 2016; Champion et al., 2019) or recurrent neural networks (RNNs) (Vlachas et al., 2018; Hernandez et al., 2020; Durstewitz, 2017;

Original: Koppe et al., 2019).
Revised: Koppe et al., 2019).

Original: Formally, RNNs are (usually discrete-time) nonlinear DS that are dynamically universal in the sense that they can approximate to arbitrary precision the ﬂow ﬁeld of any other DS on compact sets of the real space (Funahashi & Nakamura, 1993; Kimura & Nakano, 1998; Hanson & Raginsky, 2020).
Revised: Formally, RNNs are (usually discrete-time) nonlinear DS that are dynamically universal in the sense that they can approximate to arbitrary precision the ﬂow ﬁeld of any other DS on compact sets of the real space (Funahashi & Nakamura, 1993; Kimura & Nakano, 1998; Hanson & Raginsky, 2020).

Original: Hence, RNNs seem like a good choice for reconstructing – in this sense of dynamically equivalent behavior – the set of governing equations underlying real time series data.
Revised: Hence, RNNs seem like a good choice for reconstructing – in this sense of dynamically equivalent behavior – the set of governing equations underlying real time series data.

Original: However, RNNs in their vanilla form suffer from the ‘vanishing or exploding gradients’ problem (Hochreiter & Schmidhuber, 1997; Bengio et al., 1994):
Revised: However, RNNs in their vanilla form suffer from the ‘vanishing or exploding gradients’ problem (Hochreiter & Schmidhuber, 1997; Bengio et al., 1994):

Original: During training, error gradients tend to either exponentially explode or decay away across successive time steps, and hence vanilla RNNs face severe problems in capturing long time scales or long-range dependencies in the data.
Revised: During training, error gradients tend to either exponentially explode or decay away across successive time steps, and hence vanilla RNNs face severe problems in capturing long time scales or long-range dependencies in the data.

Original: Specially designed RNN architectures equipped with gating mechanisms and linear memory cells have been proposed for mitigating this issue (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).
Revised: Specially designed RNN architectures equipped with gating mechanisms and linear memory cells have been proposed for mitigating this issue (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).

Original: However, from a DST perspective, simpler models that can be more easily analyzed and interpreted in DS terms (Monfared & Durstewitz, 2020a;b), and for which more efﬁcient inference algorithms exist that emphasize approximation of the true underlying DS (Koppe et al., 2019; Hernandez et al., 2020;
Revised: However, from a DST perspective, simpler models that can be more easily analyzed and interpreted in DS terms (Monfared & Durstewitz, 2020a;b), and for which more efﬁcient inference algorithms exist that emphasize approximation of the true underlying DS (Koppe et al., 2019; Hernandez et al., 2020;

Original: Zhao & Park, 2020), would be preferable.
Revised: Zhao & Park, 2020), would be preferable.

Original: More recent solutions to the vanishing vs. exploding gradient problem attempt to retain the simplicity of vanilla RNNs by initializing or constraining the recurrent weight matrix to be the identity (Le et al., 2015), orthogonal (Henaff et al., 2016;
Revised: More recent solutions to the vanishing vs. exploding gradient problem attempt to retain the simplicity of vanilla RNNs by initializing or constraining the recurrent weight matrix to be the identity (Le et al., 2015), orthogonal (Henaff et al., 2016;

Original: Helfrich et al., 2018) or unitary (Arjovsky et al., 2016).
Revised: Helfrich et al., 2018) or unitary (Arjovsky et al., 2016).

Original: While merely initialization-based solutions, however, may be unstable and quickly dissolve during training, orthogonal or unitary constraints, on the other hand, are too restrictive for reconstructing DS, and more generally from a computational perspective as well (Kerg et al., 2019):
Revised: While merely initialization-based solutions, however, may be unstable and quickly dissolve during training, orthogonal or unitary constraints, on the other hand, are too restrictive for reconstructing DS, and more generally from a computational perspective as well (Kerg et al., 2019):

Original: For instance, neither chaotic behavior (that requires diverging directions) nor multi-stability, that is the coexistence of several distinct attractors, are possible.
Revised: For instance, neither chaotic behavior (that requires diverging directions) nor multi-stability, that is the coexistence of several distinct attractors, are possible.

Original: Here we therefore suggest a different solution to the problem which takes inspiration from computational neuroscience: Supported by experimental evidence (Daie et al., 2015; Brody et al., 2003), line or plane attractors have been suggested as a dynamical mechanism for maintaining arbitrary pieces of information in working memory (Seung, 1996; Machens et al., 2005).
Revised: Here we therefore suggest a different solution to the problem which takes inspiration from computational neuroscience: Supported by experimental evidence (Daie et al., 2015; Brody et al., 2003), line or plane attractors have been suggested as a dynamical mechanism for maintaining arbitrary pieces of information in working memory (Seung, 1996; Machens et al., 2005).

Original: 1 A line or plane attractor is a continuous set of marginally (neutrally) stable ﬁxed points to which the system’s state converges from some neighborhood, while along the line itself there is zero friction, i.e. neither con- nor divergence (Fig.
Revised: 1 A line or plane attractor is a continuous set of marginally (neutrally) stable ﬁxed points to which the system’s state converges from some neighborhood, while along the line itself there is zero friction, i.e. neither con- nor divergence (Fig.

Original: 1 A ).
Revised: 1 A ).

Original: Hence, a line attractor will perform a perfect integration of inputs and retain updated states indeﬁnitely, while a slightly detuned line attractor will equip the system with arbitrarily slow time constants (Fig. 1 B ).
Revised: Hence, a line attractor will perform a perfect integration of inputs and retain updated states indeﬁnitely, while a slightly detuned line attractor will equip the system with arbitrarily slow time constants (Fig. 1 B ).

Original: This latter conﬁguration has been suggested as a dynamical basis for interval timing and slowly ramping neural activity (Durstewitz, 2003; 2004).
Revised: This latter conﬁguration has been suggested as a dynamical basis for interval timing and slowly ramping neural activity (Durstewitz, 2003; 2004).

Original: The present idea is to exploit this dynamical setup for long short-term memory and arbitrarily slow time constants by forcing part of the RNN’s subspace toward a plane (line) attractor conﬁguration through speciﬁcally designed regularization terms.
Revised: The present idea is to exploit this dynamical setup for long short-term memory and arbitrarily slow time constants by forcing part of the RNN’s subspace toward a plane (line) attractor conﬁguration through speciﬁcally designed regularization terms.

Original: Speciﬁcally, our goal here is not so much to beat the state of the art on long short-term memory tasks, but rather to address the exploding vs. vanishing gradient problem within a simple, dynamically tractable RNN, optimized for DS reconstruction and interpretation.
Revised: Speciﬁcally, our goal here is not so much to beat the state of the art on long short-term memory tasks, but rather to address the exploding vs. vanishing gradient problem within a simple, dynamically tractable RNN, optimized for DS reconstruction and interpretation.

Original: For this we build on piecewiselinear RNNs (PLRNNs) (Koppe et al., 2019; Monfared & Durstewitz, 2020b) which employ ReLU activation functions.
Revised: For this we build on piecewiselinear RNNs (PLRNNs) (Koppe et al., 2019; Monfared & Durstewitz, 2020b) which employ ReLU activation functions.

Original: PLRNNs have a simple mathematical structure (see eq. 1) which makes them dynamically interpretable in the sense that many geometric properties of the system’s state space can in principle be computed analytically, including ﬁxed points, cycles, and their stability (Suppl.
Revised: PLRNNs have a simple mathematical structure (see eq. 1) which makes them dynamically interpretable in the sense that many geometric properties of the system’s state space can in principle be computed analytically, including ﬁxed points, cycles, and their stability (Suppl.

Original: 6.1.2; Koppe et al.
Revised: 6.1.2; Koppe et al.

Original: ; Monfared & Durstewitz (2020a)), i.e. do not require numerical techniques (Sussillo & Barak, 2013).
Revised: ; Monfared & Durstewitz (2020a)), i.e. do not require numerical techniques (Sussillo & Barak, 2013).

Original: Moreover, PLRNNs constitute a type of piecewise linear (PWL) map for which many important bifurcations have been comparatively well characterized (Monfared &
Revised: Moreover, PLRNNs constitute a type of piecewise linear (PWL) map for which many important bifurcations have been comparatively well characterized (Monfared &

Original: Durstewitz, 2020a; Avrutin et al., 2019).
Revised: Durstewitz, 2020a; Avrutin et al., 2019).

Original: PLRNNs can furthermore be translated into equivalent continuous time ordinary differential equation (ODE) systems (Monfared & Durstewitz, 2020b) which comes with further advantages for analysis, e.g. continuous ﬂow ﬁelds (Fig.
Revised: PLRNNs can furthermore be translated into equivalent continuous time ordinary differential equation (ODE) systems (Monfared & Durstewitz, 2020b) which comes with further advantages for analysis, e.g. continuous ﬂow ﬁelds (Fig.

Original: 1 A , B ).
Revised: 1 A , B ).

Original: We retain the PLRNN’s structural simplicity and analytical tractability while mitigating the exploding vs. vanishing gradient problem by adding special regularization terms for a subset of PLRNN units to the loss function.
Revised: We retain the PLRNN’s structural simplicity and analytical tractability while mitigating the exploding vs. vanishing gradient problem by adding special regularization terms for a subset of PLRNN units to the loss function.

Original: These terms are designed to push the system toward line attractor conﬁgurations, without strictly enforcing them, along some – but not all – directions in state space.
Revised: These terms are designed to push the system toward line attractor conﬁgurations, without strictly enforcing them, along some – but not all – directions in state space.

Original: We further establish a tight mathematical relationship between the PLRNN dynamics and the behavior of its gradients during training.
Revised: We further establish a tight mathematical relationship between the PLRNN dynamics and the behavior of its gradients during training.

Original: Finally, we demonstrate that our approach outperforms LSTM and other, initialization-based, methods on a number of ‘classical’ machine learning benchmarks (Hochreiter & Schmidhuber, 1997).
Revised: Finally, we demonstrate that our approach outperforms LSTM and other, initialization-based, methods on a number of ‘classical’ machine learning benchmarks (Hochreiter & Schmidhuber, 1997).

Original: Much more importantly in the present DST context, we demonstrate that our new regularization-supported inference efﬁciently captures all relevant time scales when reconstructing challenging nonlinear DS with multiple short- and long-range phenomena.
Revised: Much more importantly in the present DST context, we demonstrate that our new regularization-supported inference efﬁciently captures all relevant time scales when reconstructing challenging nonlinear DS with multiple short- and long-range phenomena.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Dynamical systems reconstruction.
Revised: Dynamical systems reconstruction.

Original: From a natural science perspective, the goal of reconstructing or identifying the underlying DS is substantially more ambitious than (and different from) building a system that ‘merely’ yields good ahead predictions: In DS identiﬁcation we require that the inferred model can freely reproduce (when no longer guided by the data) the underlying attractor geometries and state space properties (see section 3.5, Fig. S2; Kantz & Schreiber (2004)).
Revised: From a natural science perspective, the goal of reconstructing or identifying the underlying DS is substantially more ambitious than (and different from) building a system that ‘merely’ yields good ahead predictions: In DS identiﬁcation we require that the inferred model can freely reproduce (when no longer guided by the data) the underlying attractor geometries and state space properties (see section 3.5, Fig. S2; Kantz & Schreiber (2004)).

Original: [Figure]
Revised: [Figure]

Revised: add s 1 ∈

Revised: [0,1] idx s 2 ∈ {0,1}

Revised: A 1,1 =1 B 1,1 =

Original: Earlier work using RNNs for DS reconstruction (Roweis & Ghahramani, 2002; Yu et al., 2005) mainly focused on inferring the posterior over latent trajectories
Revised: W 1,2 =1 C 2,1 = C 2,2 = h 2 =– [Figure]

Original: {+z 
+}
Revised: {+z 
+}

Revised: = z

Revised: [Figure]

Revised: Earlier work using RNNs for DS reconstruction (Roweis & Ghahramani, 2002; Yu et al., 2005) mainly focused on inferring the posterior over latent trajectories

Original: Z
Revised: Z

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: , z T } given time series data X = { x 1 , . . .
Revised: , z T } given time series data X = { x 1 , . . .

Original: , x T } , p ( Z | X ) , and on ahead predictions (Lu et al., 2017), as does much of the recent work on variational inference of DS (Duncker et al., 2019; Zhao & Park, 2020; Hernandez et al., 2020).
Revised: , x T } , p ( Z | X ) , and on ahead predictions (Lu et al., 2017), as does much of the recent work on variational inference of DS (Duncker et al., 2019; Zhao & Park, 2020; Hernandez et al., 2020).

Original: Although this enables insight into the dynamics along the empirically observed trajectories, both – posterior inference and good ahead predictions – do not per se guarantee that the inferred models can generate the underlying attractor geometries on their own (see Fig.
Revised: Although this enables insight into the dynamics along the empirically observed trajectories, both – posterior inference and good ahead predictions – do not per se guarantee that the inferred models can generate the underlying attractor geometries on their own (see Fig.

Original: S2, Koppe et al.
Revised: S2, Koppe et al.

Original: In contrast, if fully generative reconstruction of the underlying DS in this latter sense were achieved, formal analysis or simulation of the resulting RNN equations could provide a much deeper understanding of the dynamical mechanisms underlying empirical observations (Fig. 1 C ).
Revised: In contrast, if fully generative reconstruction of the underlying DS in this latter sense were achieved, formal analysis or simulation of the resulting RNN equations could provide a much deeper understanding of the dynamical mechanisms underlying empirical observations (Fig. 1 C ).

Original: Some approaches geared toward this latter goal of full DS reconstruction make speciﬁc structural assumptions about the form of the DS equations (‘white box approach’; Meeds et al.
Revised: Some approaches geared toward this latter goal of full DS reconstruction make speciﬁc structural assumptions about the form of the DS equations (‘white box approach’; Meeds et al.

Original: ; Raissi (2018);
Revised: ; Raissi (2018);

Original: Gorbach et al.
Revised: Gorbach et al.

Original: (2017)), e.g. based on physical or biological domain knowledge, and focus on estimating the system’s latent states and parameters, rather than approximating an unknown DS based on the observed time series information alone (‘black box approach’).
Revised: (2017)), e.g. based on physical or biological domain knowledge, and focus on estimating the system’s latent states and parameters, rather than approximating an unknown DS based on the observed time series information alone (‘black box approach’).

Original: Others (Trischler &
Revised: Others (Trischler &

Original: D’Eleuterio, 2016; Brunton et al., 2016; Champion et al., 2019) attempt to approximate the ﬂow ﬁeld, obtained e.g. by numerical differentiation, directly through basis expansions or neural networks.
Revised: D’Eleuterio, 2016; Brunton et al., 2016; Champion et al., 2019) attempt to approximate the ﬂow ﬁeld, obtained e.g. by numerical differentiation, directly through basis expansions or neural networks.

Original: However, numerical derivatives are problematic for their high variance and other numerical issues (Raissi, 2018;
Revised: However, numerical derivatives are problematic for their high variance and other numerical issues (Raissi, 2018;

Original: Baydin et al., 2018; Chen et al., 2017).
Revised: Baydin et al., 2018; Chen et al., 2017).

Original: Another factor to consider is that in many biological systems like the brain the intrinsic dynamics are highly stochastic with many noise sources, like probabilistic synaptic release (Stevens, 2003).
Revised: Another factor to consider is that in many biological systems like the brain the intrinsic dynamics are highly stochastic with many noise sources, like probabilistic synaptic release (Stevens, 2003).

Original: Models that do not explicitly account for dynamical process noise (Ayed et al., 2019; Champion et al., 2019; Rudy et al., 2019) are therefore less suited and more vulnerable to model misspeciﬁcation.
Revised: Models that do not explicitly account for dynamical process noise (Ayed et al., 2019; Champion et al., 2019; Rudy et al., 2019) are therefore less suited and more vulnerable to model misspeciﬁcation.

Original: Finally, some fully probabilistic models for DS reconstruction based on GRU (Fraccaro et al., 2016), LSTM (Zheng et al., 2017; Vlachas et al., 2018), or radial basis function (Zhao & Park, 2020) networks, are not easily interpretable and amenable to DS analysis in the sense deﬁned in sect.
Revised: Finally, some fully probabilistic models for DS reconstruction based on GRU (Fraccaro et al., 2016), LSTM (Zheng et al., 2017; Vlachas et al., 2018), or radial basis function (Zhao & Park, 2020) networks, are not easily interpretable and amenable to DS analysis in the sense deﬁned in sect.

Original: Most importantly, none of these previous approaches consider the long-range dependency problem within more easily tractable RNNs for DS.
Revised: Most importantly, none of these previous approaches consider the long-range dependency problem within more easily tractable RNNs for DS.

Original: Long-range dependency problems in RNNs .
Revised: Long-range dependency problems in RNNs .

Original: Error gradients in vanilla RNNs tend to either explode or vanish due to the large product of derivative terms that results from recursive application of the chain rule over time steps (Hochreiter, 1991; Bengio et al., 1994; Hochreiter & Schmidhuber, 1997).
Revised: Error gradients in vanilla RNNs tend to either explode or vanish due to the large product of derivative terms that results from recursive application of the chain rule over time steps (Hochreiter, 1991; Bengio et al., 1994; Hochreiter & Schmidhuber, 1997).

Original: To address this issue, RNNs with gated memory cells (Hochreiter & Schmidhuber, 1997; Cho et al., 2014) have been speciﬁcally designed, but their more complicated mathematical structure makes them less amenable to a systematic DS analysis.
Revised: To address this issue, RNNs with gated memory cells (Hochreiter & Schmidhuber, 1997; Cho et al., 2014) have been speciﬁcally designed, but their more complicated mathematical structure makes them less amenable to a systematic DS analysis.

Original: Even simple objects like ﬁxed points of these systems have to be found by numerical techniques (Sussillo & Barak, 2013; Jordan et al., 2019).
Revised: Even simple objects like ﬁxed points of these systems have to be found by numerical techniques (Sussillo & Barak, 2013; Jordan et al., 2019).

Original: Thus, approaches which retain the simplicity of vanilla RNNs while solving the exploding vs. vanishing gradients problem would be desirable.
Revised: Thus, approaches which retain the simplicity of vanilla RNNs while solving the exploding vs. vanishing gradients problem would be desirable.

Original: Recently, Le et al.
Revised: Recently, Le et al.

Original: (2015) observed that initialization of the recurrent weight matrix W to the identity in ReLU-based RNNs may yield performance en par with
Revised: (2015) observed that initialization of the recurrent weight matrix W to the identity in ReLU-based RNNs may yield performance en par with

Original: LSTMs on standard machine learning benchmarks.
Revised: LSTMs on standard machine learning benchmarks.

Original: Talathi & Vartak (2016) expanded on this idea by initializing the recurrence matrix such that its largest absolute eigenvalue is 1.
Revised: Talathi & Vartak (2016) expanded on this idea by initializing the recurrence matrix such that its largest absolute eigenvalue is 1.

Original: Later work enforced orthogonal (Henaff et al., 2016;
Revised: Later work enforced orthogonal (Henaff et al., 2016;

Original: Helfrich et al., 2018;
Revised: Helfrich et al., 2018;

Original: Jing et al., 2019) or unitary (Arjovsky et al., 2016) constraints on the recurrent weight matrix during training.
Revised: Jing et al., 2019) or unitary (Arjovsky et al., 2016) constraints on the recurrent weight matrix during training.

Original: While this appears to yield long-term memory performance sometimes superior to that of LSTMs (but see (Henaff et al., 2016)), these networks are limited in their computational power (Kerg et al., 2019).
Revised: While this appears to yield long-term memory performance sometimes superior to that of LSTMs (but see (Henaff et al., 2016)), these networks are limited in their computational power (Kerg et al., 2019).

Original: This may be a consequence of the fact that RNNs with orthogonal recurrence matrix are quite restricted in the range of dynamical phenomena they can produce, e.g. chaotic attractors are not possible since (locally) diverging eigen-directions are disabled.
Revised: This may be a consequence of the fact that RNNs with orthogonal recurrence matrix are quite restricted in the range of dynamical phenomena they can produce, e.g. chaotic attractors are not possible since (locally) diverging eigen-directions are disabled.

Original: Our approach therefore is to establish line/plane attractors only along some but not all directions in state space, and to only push the RNN toward these conﬁgurations but not strictly enforce them, such that convergence or (local) divergence of RNN dynamics is still possible.
Revised: Our approach therefore is to establish line/plane attractors only along some but not all directions in state space, and to only push the RNN toward these conﬁgurations but not strictly enforce them, such that convergence or (local) divergence of RNN dynamics is still possible.

Original: We furthermore implement these concepts through regularization terms in the loss functions, rather than through mere initialization.
Revised: We furthermore implement these concepts through regularization terms in the loss functions, rather than through mere initialization.

Original: This way plane attractors are encouraged throughout training without fading away.
Revised: This way plane attractors are encouraged throughout training without fading away.

Original:  M ODEL FORMULATION AND THEORETICAL ANALYSIS 3. B ASIC MODEL FORMULATION
Revised:  M ODEL FORMULATION AND THEORETICAL ANALYSIS 3. B ASIC MODEL FORMULATION

Original: Assume we are given two multivariate time series S
Revised: Assume we are given two multivariate time series S

Original: = { s t } and X = { x t } , one we will denote as ‘inputs’ ( S ) and the other as ‘outputs’ ( X ).
Revised: = { s t } and X = { x t } , one we will denote as ‘inputs’ ( S ) and the other as ‘outputs’ ( X ).

Original: In the ‘classical’ ( supervised ) machine learning setting, we usually wish to map S on X through a RNN with latent state equation z t =
Revised: In the ‘classical’ ( supervised ) machine learning setting, we usually wish to map S on X through a RNN with latent state equation z t =

Original: F θ ( z t − 1 , s t ) and outputs x t ∼ p λ ( x t | z t ) , as for instance in the ‘addition problem’ (Hochreiter & Schmidhuber, 1997).
Revised: F θ ( z t − 1 , s t ) and outputs x t ∼ p λ ( x t | z t ) , as for instance in the ‘addition problem’ (Hochreiter & Schmidhuber, 1997).

Original: In DS reconstruction, in contrast, we usually have a dense time series X from which we wish to infer ( unsupervised ) the underlying DS, where S may provide an additional forcing function or sparse experimental inputs or perturbations.
Revised: In DS reconstruction, in contrast, we usually have a dense time series X from which we wish to infer ( unsupervised ) the underlying DS, where S may provide an additional forcing function or sparse experimental inputs or perturbations.

Original: While our focus in this paper is on this latter task, DS reconstruction, we will demonstrate that our approach brings beneﬁts in both these settings.
Revised: While our focus in this paper is on this latter task, DS reconstruction, we will demonstrate that our approach brings beneﬁts in both these settings.

Original: The latent RNN we consider here is a piecewise-linear RNN (PLRNN) which takes the speciﬁc form (Koppe et al., 2019)
Revised: The latent RNN we consider here is a piecewise-linear RNN (PLRNN) which takes the speciﬁc form (Koppe et al., 2019)

Original: [Equation]
Revised: [Equation]

Original: where z t ∈ R M × 1 is the hidden state (column) vector of dimension M , A ∈ R M × M a diagonal and
Revised: where z t ∈ R M × 1 is the hidden state (column) vector of dimension M , A ∈ R M × M a diagonal and

Original: W ∈ R M × M an off-diagonal matrix, s t ∈ R K × 1 the external input of dimension K , C ∈ R M × K the input mapping, h ∈ R M × 1 a bias, and ε t a Gaussian noise term with diagonal covariance matrix diag( Σ ) ∈ R M + .
Revised: W ∈ R M × M an off-diagonal matrix, s t ∈ R K × 1 the external input of dimension K , C ∈ R M × K the input mapping, h ∈ R M × 1 a bias, and ε t a Gaussian noise term with diagonal covariance matrix diag( Σ ) ∈ R M + .

Original: The nonlinearity φ ( z ) is a ReLU, φ ( z ) i = max(0 , z i ) , i ∈ { 1 , . . .
Revised: The nonlinearity φ ( z ) is a ReLU, φ ( z ) i = max(0 , z i ) , i ∈ { 1 , . . .

Original: , M } .
Revised: , M } .

Original: This speciﬁc formulation represents a discrete-time version of ﬁring rate (population) models as used in computational neuroscience (Song et al., 2016; Durstewitz, 2017;
Revised: This speciﬁc formulation represents a discrete-time version of ﬁring rate (population) models as used in computational neuroscience (Song et al., 2016; Durstewitz, 2017;

Original: Engelken et al., 2020).
Revised: Engelken et al., 2020).

Original: We will assume that the latent RNN states z t are coupled to the actual observations x t through a simple observation model of the form
Revised: We will assume that the latent RNN states z t are coupled to the actual observations x t through a simple observation model of the form

Original: [Equation]
Revised: [Equation]

Original: in the case of observations x t ∈ R N × 1 , where B ∈ R N × M is a factor loading matrix, g some (usually monotonic) nonlinear transfer function (e.g., ReLU), and diag( Γ ) ∈ R N + the diagonal covariance matrix of the Gaussian observation noise, or through a softmax function in case of categorical observations x i,t ∈ { 0 , 1 } (see Suppl.
Revised: in the case of observations x t ∈ R N × 1 , where B ∈ R N × M is a factor loading matrix, g some (usually monotonic) nonlinear transfer function (e.g., ReLU), and diag( Γ ) ∈ R N + the diagonal covariance matrix of the Gaussian observation noise, or through a softmax function in case of categorical observations x i,t ∈ { 0 , 1 } (see Suppl.

Original: 6.1.7 for details).
Revised: 6.1.7 for details).

Original: 3. R EGULARIZATION APPROACH
Revised: 3. R EGULARIZATION APPROACH

Original: First note that by letting A
Revised: First note that by letting A

Original: = I , W = 0 , and h = 0 in eq.
Revised: = I , W = 0 , and h = 0 in eq.

Original: 1, every point in z space will be a marginally stable ﬁxed point of the system, leading it to perform a perfect integration of external inputs as in parametric working memory (Machens et al., 2005; Brody et al., 2003).
Revised: 1, every point in z space will be a marginally stable ﬁxed point of the system, leading it to perform a perfect integration of external inputs as in parametric working memory (Machens et al., 2005; Brody et al., 2003).

Original: This is similar in spirit to Le et al.
Revised: This is similar in spirit to Le et al.

Original: (2015) who initialized RNN parameters such that it performs an identity mapping for z i,t ≥ 0 .
Revised: (2015) who initialized RNN parameters such that it performs an identity mapping for z i,t ≥ 0 .

Original: However, here 1) we use a neuroscientiﬁcally motivated network architecture (eq. 1) that enables the identity mapping across the variables’ entire support , z i,t ∈ [ −∞ , + ∞ ] , which we conjecture will be of advantage for establishing long short-term memory properties, 2) we encourage this mapping only for a subset M reg ≤ M of units (Fig. S1), leaving others free to perform arbitrary computations, and 3) we stabilize this conﬁguration throughout training by introducing a speciﬁc
Revised: However, here 1) we use a neuroscientiﬁcally motivated network architecture (eq. 1) that enables the identity mapping across the variables’ entire support , z i,t ∈ [ −∞ , + ∞ ] , which we conjecture will be of advantage for establishing long short-term memory properties, 2) we encourage this mapping only for a subset M reg ≤ M of units (Fig. S1), leaving others free to perform arbitrary computations, and 3) we stabilize this conﬁguration throughout training by introducing a speciﬁc

Original: L 2 regularization for parameters
Revised: L 2 regularization for parameters

Original: A , W , and h in eq. 1.
Revised: A , W , and h in eq. 1.

Original: When embedded into a larger, convergent system, we will call this conﬁguration more generally a manifold attractor .
Revised: When embedded into a larger, convergent system, we will call this conﬁguration more generally a manifold attractor .

Original: That way, we divide the units into two types, where the regularized units serve as a memory that tends to decay very slowly (depending on the size of the regularization term), while the remaining units maintain the ﬂexibility to approximate any underlying DS, yet retaining the simplicity of the original PLRNN (eq. 1).
Revised: That way, we divide the units into two types, where the regularized units serve as a memory that tends to decay very slowly (depending on the size of the regularization term), while the remaining units maintain the ﬂexibility to approximate any underlying DS, yet retaining the simplicity of the original PLRNN (eq. 1).

Original: Speciﬁcally, the following penalty is added to the loss function (Fig. S1):
Revised: Speciﬁcally, the following penalty is added to the loss function (Fig. S1):

Original: [Equation]
Revised: [Equation]

Original: (Recall from sect.
Revised: (Recall from sect.

Original: 3.1 that A is a diagonal and W is an off-diagonal matrix.)
Revised: 3.1 that A is a diagonal and W is an off-diagonal matrix.)

Original: While this formulation allows us to trade off, for instance, the tendency toward a manifold attractor ( A → I , h → 0 ) vs. the sensitivity to other units’ inputs ( W → 0 ), for all experiments performed here a common value, τ A = τ W = τ h
Revised: While this formulation allows us to trade off, for instance, the tendency toward a manifold attractor ( A → I , h → 0 ) vs. the sensitivity to other units’ inputs ( W → 0 ), for all experiments performed here a common value, τ A = τ W = τ h

Original: = τ , was assumed for the three regularization factors.
Revised: = τ , was assumed for the three regularization factors.

Original: We will refer to ( z 1 . . .
Revised: We will refer to ( z 1 . . .

Original: z M reg ) as the regularized (‘memory’) subsystem, and to ( z M reg +1 . . .
Revised: z M reg ) as the regularized (‘memory’) subsystem, and to ( z M reg +1 . . .

Original: z M ) as the non-regularized (‘computational’) subsystem.
Revised: z M ) as the non-regularized (‘computational’) subsystem.

Original: Note that in the limit τ → ∞ exact manifold attractors will be enforced.
Revised: Note that in the limit τ → ∞ exact manifold attractors will be enforced.

Original: 3. T HEORETICAL ANALYSIS
Revised: 3. T HEORETICAL ANALYSIS

Original: We will now establish a tight connection between the PLRNN dynamics and its error gradients.
Revised: We will now establish a tight connection between the PLRNN dynamics and its error gradients.

Original: Similar ideas appeared in Chang et al.
Revised: Similar ideas appeared in Chang et al.

Original: , but these authors focused only on ﬁxed point dynamics, while here we will consider the more general case including cycles of any order.
Revised: , but these authors focused only on ﬁxed point dynamics, while here we will consider the more general case including cycles of any order.

Original: First, note that by interpretability of model eq.
Revised: First, note that by interpretability of model eq.

Original: 1 we mean that it is easily amenable to a rigorous DS analysis: As shown in Suppl.
Revised: 1 we mean that it is easily amenable to a rigorous DS analysis: As shown in Suppl.

Original: 6.1.2, we can explicitly determine all the system’s ﬁxed points and cycles and their stability.
Revised: 6.1.2, we can explicitly determine all the system’s ﬁxed points and cycles and their stability.

Original: Moreover, as shown in Monfared & Durstewitz (2020b), we can – under certain conditions – transform the PLRNN into an equivalent continuous-time (ODE) piecewise-linear system, which brings further advantages for DS analysis.
Revised: Moreover, as shown in Monfared & Durstewitz (2020b), we can – under certain conditions – transform the PLRNN into an equivalent continuous-time (ODE) piecewise-linear system, which brings further advantages for DS analysis.

Original: Let us rewrite eq.
Revised: Let us rewrite eq.

Original: 1 in the form
Revised: 1 in the form

Original: [Equation]
Revised: [Equation]

Original: where D Ω( t − 1) is the diagonal matrix of outer derivatives of the ReLU function evaluated at z t −(see Suppl.
Revised: where D Ω( t − 1) is the diagonal matrix of outer derivatives of the ReLU function evaluated at z t −(see Suppl.

Original: 6.1.2), and we ignore external inputs and noise terms for now.
Revised: 6.1.2), and we ignore external inputs and noise terms for now.

Original: Starting from some initial condition z 1 , we can recursively develop z T as (see Suppl.
Revised: Starting from some initial condition z 1 , we can recursively develop z T as (see Suppl.

Original: 6.1.2 for more details):
Revised: 6.1.2 for more details):

Original: [Equation]
Revised: [Equation]

Original: Likewise, for some common loss function L ( A , W , h ) =
Revised: Likewise, for some common loss function L ( A , W , h ) =

Original: (cid:80) Tt =
Revised: (cid:80) Tt =

Original: 2 L t , we can recursively develop the derivatives w.r.t.
Revised: 2 L t , we can recursively develop the derivatives w.r.t.

Original: weights w mk (and similar for components of A and h ) as
Revised: weights w mk (and similar for components of A and h ) as

Original: [Equation]
Revised: [Equation]

Original: where 1 ( m,k ) is an M × M indicator matrix with a 1 for the ( m, k ) ’
Revised: where 1 ( m,k ) is an M × M indicator matrix with a 1 for the ( m, k ) ’

Original: th entry and 0 everywhere else.
Revised: th entry and 0 everywhere else.

Original: Observing that eqs.
Revised: Observing that eqs.

Original: 5 and 6 contain similar product terms which determine the system’s long-term behavior, our ﬁrst theorem links the PLRNN dynamics to its total error gradients:
Revised: 5 and 6 contain similar product terms which determine the system’s long-term behavior, our ﬁrst theorem links the PLRNN dynamics to its total error gradients:

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider a PLRNN given by eq. 4, and assume that it converges to a stable ﬁxed point, say z t ∗ 1 :=
Revised: Consider a PLRNN given by eq. 4, and assume that it converges to a stable ﬁxed point, say z t ∗ 1 :=

Original: z ∗ 1 , or a k -cycle ( k > 1) with the periodic points { z t ∗ k , z t ∗ k − 1 , · · · , z t ∗ k − ( k − 1) } , for
Revised: z ∗ 1 , or a k -cycle ( k > 1) with the periodic points { z t ∗ k , z t ∗ k − 1 , · · · , z t ∗ k − ( k − 1) } , for

Original: T → ∞ .
Revised: T → ∞ .

Original: Suppose that, for k ≥ 1 and i ∈ { 0 , 1 , · · · , k − 1 } , σ max ( W Ω( t ∗ k − i ) ) =
Revised: Suppose that, for k ≥ 1 and i ∈ { 0 , 1 , · · · , k − 1 } , σ max ( W Ω( t ∗ k − i ) ) =

Original: (cid:13)(cid:13) W Ω( t ∗ k − i ) (cid:13)(cid:13) < 1 , where W Ω( t ∗ k − i ) denotes the Jacobian of the system at z t ∗ k − i and σ max indicates the largest singular value of a matrix.
Revised: (cid:13)(cid:13) W Ω( t ∗ k − i ) (cid:13)(cid:13) < 1 , where W Ω( t ∗ k − i ) denotes the Jacobian of the system at z t ∗ k − i and σ max indicates the largest singular value of a matrix.

Original: Then, the 2 -norms of the tensors collecting all derivatives, (cid:13)(cid:13) ∂
Revised: Then, the 2 -norms of the tensors collecting all derivatives, (cid:13)(cid:13) ∂

Original: z T ∂ W (cid:13)(cid:13) 2 , (cid:13)(cid:13) ∂
Revised: z T ∂ W (cid:13)(cid:13) 2 , (cid:13)(cid:13) ∂

Original: z T ∂ A (cid:13)(cid:13) 2 , (cid:13)(cid:13) ∂
Revised: z T ∂ A (cid:13)(cid:13) 2 , (cid:13)(cid:13) ∂

Original: z T ∂ h
Revised: z T ∂ h

Original: (cid:13)(cid:13) 2 , will be bounded from above, i.e. will not diverge for T → ∞ .
Revised: (cid:13)(cid:13) 2 , will be bounded from above, i.e. will not diverge for T → ∞ .

Original: Proof.
Revised: Proof.

Original: See Suppl.
Revised: See Suppl.

Original: sect.
Revised: sect.

Original: 6.1 (subsection 6.1.3).
Revised: 6.1 (subsection 6.1.3).

Original: (cid:3)
Revised: (cid:3)

Original: While Theorem 1 is a general statement about PLRNN dynamics and total gradients, our next theorem more speciﬁcally provides conditions under which Jacobians linking temporally distant states z T and z t , T (cid:29) t , will neither vanish nor explode in the regularized PLRNN:
Revised: While Theorem 1 is a general statement about PLRNN dynamics and total gradients, our next theorem more speciﬁcally provides conditions under which Jacobians linking temporally distant states z T and z t , T (cid:29) t , will neither vanish nor explode in the regularized PLRNN:

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume a PLRNN with matrix A + W partitioned as in Fig.
Revised: Assume a PLRNN with matrix A + W partitioned as in Fig.

Original: S1, i.e. with the ﬁrst
Revised: S1, i.e. with the ﬁrst

Original: M reg rows corresponding to those of an M × M identity matrix.
Revised: M reg rows corresponding to those of an M × M identity matrix.

Original: Suppose that the non-regularized subsystem ( z M reg +1 . . .
Revised: Suppose that the non-regularized subsystem ( z M reg +1 . . .

Original: z M ) , if considered in isolation, satisﬁes Theorem 1, i.e. converges to a kcycle with k ≥ 1 .
Revised: z M ) , if considered in isolation, satisﬁes Theorem 1, i.e. converges to a kcycle with k ≥ 1 .

Original: Then, for the full system ( z 1 . . .
Revised: Then, for the full system ( z 1 . . .

Original: z M ) , the 2-norm of the Jacobians connecting temporally distal states z T and z t will be bounded from above and below for all T > t , i.e. ∞ > ρ up ≥ (cid:13) (cid:13)(cid:13)
Revised: z M ) , the 2-norm of the Jacobians connecting temporally distal states z T and z t will be bounded from above and below for all T > t , i.e. ∞ > ρ up ≥ (cid:13) (cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: z T ∂ z t (cid:13) (cid:13)(cid:13) 

Revised: z T ∂ z t (cid:13) (cid:13)(cid:13)

Original: = (cid:13) (cid:13)(cid:13)(cid:81) t<k ≤ T W Ω( k ) (cid:13) (cid:13)(cid:13) 2 ≥
Revised: = (cid:13) (cid:13)(cid:13)(cid:81) t<k ≤ T W Ω( k ) (cid:13) (cid:13)(cid:13) 2 ≥

Original: ρ low > 0 .
Revised: ρ low > 0 .

Original: In particular, for state variables z iT and z jt such that i ∈
Revised: In particular, for state variables z iT and z jt such that i ∈

Original: { M reg + 1 , · · · , M } and j ∈ { 1 , · · · , M reg } , i.e. that connect states from the ‘memory’ to those of the ‘computational’ subsystem, one also has ∞ > λ up ≥
Revised: { M reg + 1 , · · · , M } and j ∈ { 1 , · · · , M reg } , i.e. that connect states from the ‘memory’ to those of the ‘computational’ subsystem, one also has ∞ > λ up ≥

Original: [Table]
Revised: [Table]

Original: ∂z iT ∂z
Revised: ∂z iT ∂z

Original: [Table]
Revised: [Table]

Original: ≥ λ low >as T − t → ∞ , i.e. these derivatives will never vanish nor explode.
Revised: ≥ λ low >as T − t → ∞ , i.e. these derivatives will never vanish nor explode.

Original: Proof.
Revised: Proof.

Original: See Suppl.
Revised: See Suppl.

Original: sect.
Revised: sect.

Original: 6.1 (subsection 6.1.4).
Revised: 6.1 (subsection 6.1.4).

Original: (cid:3)
Revised: (cid:3)

Original: The bounds ρ up , ρ low , λ up , λ low , are given in Suppl.
Revised: The bounds ρ up , ρ low , λ up , λ low , are given in Suppl.

Original: sect.
Revised: sect.

Original: We remark that when the regularization conditions are not exactly met, i.e. when parameters A and W slightly deviate from those in
Revised: We remark that when the regularization conditions are not exactly met, i.e. when parameters A and W slightly deviate from those in

Original: Fig.
Revised: Fig.

Original: S1, memory and gradients may ultimately dissipate, but only very slowly, as actually required for temporal processes with very long yet not inﬁnite time constants.
Revised: S1, memory and gradients may ultimately dissipate, but only very slowly, as actually required for temporal processes with very long yet not inﬁnite time constants.

Original: 3. T RAINING PROCEDURES
Revised: 3. T RAINING PROCEDURES

Original: For the (supervised) machine learning problems, all networks were trained by stochastic gradient descent (SGD) to minimize the squared-error loss between estimated and actual outputs for the addition and multiplication problems, and the cross entropy loss for sequential MNIST (see Suppl.
Revised: For the (supervised) machine learning problems, all networks were trained by stochastic gradient descent (SGD) to minimize the squared-error loss between estimated and actual outputs for the addition and multiplication problems, and the cross entropy loss for sequential MNIST (see Suppl.

Original: Adam (Kingma & Ba, 2014) from PyTorch package (Paszke et al., 2017) was used as the optimizer, with a learning rate of 0.001, gradient clip parameter of 10, and batch size of 500.
Revised: Adam (Kingma & Ba, 2014) from PyTorch package (Paszke et al., 2017) was used as the optimizer, with a learning rate of 0.001, gradient clip parameter of 10, and batch size of 500.

Original: SGD was stopped after 100 epochs and the ﬁt with the lowest loss across all epochs was taken, except for LSTM which was allowed to run for up to 200 epochs as it took longer to converge (Fig. S10).
Revised: SGD was stopped after 100 epochs and the ﬁt with the lowest loss across all epochs was taken, except for LSTM which was allowed to run for up to 200 epochs as it took longer to converge (Fig. S10).

Original: For comparability, the PLRNN latent state dynamics eq. 1 was assumed to be deterministic in this setting (i.e., Σ = 0 ), g ( z t ) =
Revised: For comparability, the PLRNN latent state dynamics eq. 1 was assumed to be deterministic in this setting (i.e., Σ = 0 ), g ( z t ) =

Original: z t and Γ =
Revised: z t and Γ =

Original: I N in eq.
Revised: I N in eq.

Original: For the regularized PLRNN (rPLRNN), penalty eq. 3 was added to the loss function.
Revised: For the regularized PLRNN (rPLRNN), penalty eq. 3 was added to the loss function.

Original: For the (unsupervised) DS reconstruction problems, the fully probabilistic, generative RNN eq.
Revised: For the (unsupervised) DS reconstruction problems, the fully probabilistic, generative RNN eq.

Original: 1 was considered.
Revised: 1 was considered.

Original: Together with eq.
Revised: Together with eq.

Original: 2 (where we take g ( z t ) =
Revised: 2 (where we take g ( z t ) =

Original: φ ( z t ))
Revised: φ ( z t ))

Original: this gives the typical form of a nonlinear state space model (Durbin & Koopman, 2012) with observation and process noise, and an Expectation-Maximization (EM) algorithm that efﬁciently exploits the model’s piecewise linear structure (Durstewitz, 2017; Koppe et al., 2019) was used to solve for the parameters by maximum likelihood.
Revised: this gives the typical form of a nonlinear state space model (Durbin & Koopman, 2012) with observation and process noise, and an Expectation-Maximization (EM) algorithm that efﬁciently exploits the model’s piecewise linear structure (Durstewitz, 2017; Koppe et al., 2019) was used to solve for the parameters by maximum likelihood.

Original: Details are given in Suppl. 6.1.5.
Revised: Details are given in Suppl. 6.1.5.

Original: All code used here will be made openly available at ***.
Revised: All code used here will be made openly available at https://github.com/DurstewitzLab/regularized-PLRNN.

Original: 3. P ERFORMANCE MEASURES
Revised: 3. P ERFORMANCE MEASURES

Original: For the machine learning benchmarks we employed the same criteria as used for optimization (MSE or cross-entropy, Suppl. 6.1.7) as performance metrics, evaluated across left-out test sets.
Revised: For the machine learning benchmarks we employed the same criteria as used for optimization (MSE or cross-entropy, Suppl. 6.1.7) as performance metrics, evaluated across left-out test sets.

Original: In addition, we report the relative frequency P correct of correctly predicted trials across the test set (see
Revised: In addition, we report the relative frequency P correct of correctly predicted trials across the test set (see

Original: Suppl.
Revised: Suppl.

Original: 6.1.7 for details).
Revised: 6.1.7 for details).

Original: For DS reconstruction problems, it is not sufﬁcient or even sensible to judge a method’s ability to infer the underlying DS purely based on some form of (ahead-)prediction error like the MSE deﬁned on the time series itself (Ch.12 in Kantz & Schreiber (2004)).
Revised: For DS reconstruction problems, it is not sufﬁcient or even sensible to judge a method’s ability to infer the underlying DS purely based on some form of (ahead-)prediction error like the MSE deﬁned on the time series itself (Ch.12 in Kantz & Schreiber (2004)).

Original: Rather, we require that the inferred model can freely reproduce (when no longer guided by the data) the underlying attractor geometries and state space properties.
Revised: Rather, we require that the inferred model can freely reproduce (when no longer guided by the data) the underlying attractor geometries and state space properties.

Original: This is not automatically guaranteed for a model yields agreeable ahead predictions on 
Revised: This is not automatically guaranteed for a model yields agreeable ahead predictions on a

Original: (Fig.
Revised: (Fig.

Original: S2 Acf.
Revised: S2 A

Original: Koppe et al.
Revised: Koppe et al.

Original: Wood (2010)).
Revised: Wood (2010)).

Original: We therefore followed Koppe et al.
Revised: We therefore followed Koppe et al.

Original: and used the Kullback-Leibler divergence between true and reproduced probability distributions across states in state space to quantify how well an inferred PLRNN captured the underlying dynamics, thus assessing the agreement in attractor geometries (cf.
Revised: and used the Kullback-Leibler divergence between true and reproduced probability distributions across states in state space to quantify how well an inferred PLRNN captured the underlying dynamics, thus assessing the agreement in attractor geometries (cf.

Original: Takens (1981); Sauer et al.
Revised: Takens (1981); Sauer et al.

Original: (see Suppl.
Revised: (see Suppl.

Original: 6.1.6 for more details).
Revised: 6.1.6 for more details).

Original:  N UMERICAL EXPERIMENTS 4. M ACHINE LEARNING BENCHMARKS
Revised:  N UMERICAL EXPERIMENTS 4. M ACHINE LEARNING BENCHMARKS

Original: Although not our prime interest here, we ﬁrst examined how the rPLRNN would fare on supervised machine learning benchmarks where inputs ( S ) are to be mapped onto target outputs ( X ) across long time spans (i.e., requiring long short-term maintenance of information), namely the addition and multiplication problems (Talathi & Vartak, 2016;
Revised: Although not our prime interest here, we ﬁrst examined how the rPLRNN would fare on supervised machine learning benchmarks where inputs ( S ) are to be mapped onto target outputs ( X ) across long time spans (i.e., requiring long short-term maintenance of information), namely the addition and multiplication problems (Talathi & Vartak, 2016;

Original: Hochreiter & Schmidhuber, 1997), and sequential
Revised: Hochreiter & Schmidhuber, 1997), and sequential

Original: MNIST (LeCun et al., 2010).
Revised: MNIST (LeCun et al., 2010).

Original: Details of these experimental setups are in Suppl.
Revised: Details of these experimental setups are in Suppl.

Original: Performance of the rPLRNN (eq. 1, eq. 3) on all 3 benchmarks was compared to several other models summarized in Suppl.
Revised: Performance of the rPLRNN (eq. 1, eq. 3) on all 3 benchmarks was compared to several other models summarized in Suppl.

Original: Table 1.
Revised: Table 1.

Original: To achieve a meaningful comparison, all models have the same number M =(based on Fig.
Revised: To achieve a meaningful comparison, all models have the same number M =(based on Fig.

Original: S3) of hidden states (which gives LSTMs overall about 4 times as many trainable parameters).
Revised: S3) of hidden states (which gives LSTMs overall about 4 times as many trainable parameters).

Original: On all three problems the rPLRNN outperforms all other tested methods, including
Revised: On all three problems the rPLRNN outperforms all other tested methods, including

Original: LSTM, iRNN (RNN initialized by the identity matrix as in Le et al.
Revised: LSTM, iRNN (RNN initialized by the identity matrix as in Le et al.

Original: (2015)), and a version of the orthogonal RNN (oRNN; Vorontsov et al.
Revised: (2015)), and a version of the orthogonal RNN (oRNN; Vorontsov et al.

Original: (similar results were obtained for other settings of
Revised: (similar results were obtained for other settings of

Original: M and batch size).
Revised: M and batch size).

Original: LSTM performs even worse than iRNN and iPLRNN (PLRNN initialized with the identity as the iRNN), although it had 4 times as many parameters and was given twice as many epochs (and thus opportunities) for training, as it also took longer to converge (Fig. S10).
Revised: LSTM performs even worse than iRNN and iPLRNN (PLRNN initialized with the identity as the iRNN), although it had 4 times as many parameters and was given twice as many epochs (and thus opportunities) for training, as it also took longer to converge (Fig. S10).

Original: In addition, the iPLRNN tends to perform slightly better than the iRNN on all three problems, suggesting that the speciﬁc structure eq.
Revised: In addition, the iPLRNN tends to perform slightly better than the iRNN on all three problems, suggesting that the speciﬁc structure eq.

Original: 1 of the PLRNN that allows for a manifold attractor across the variables’ full range may be advantageous to begin with, while the regularization further improves performance.
Revised: 1 of the PLRNN that allows for a manifold attractor across the variables’ full range may be advantageous to begin with, while the regularization further improves performance.

Original: 4. N UMERICAL EXPERIMENTS ON DYNAMICAL SYSTEMS WITH DIFFERENT TIME SCALES
Revised: 4. N UMERICAL EXPERIMENTS ON DYNAMICAL SYSTEMS WITH DIFFERENT TIME SCALES

Original: While it is encouraging that the rPLRNN may perform even better than several previous approaches to the vanishing vs. exploding gradients problem, our major goal here was to examine whether our regularization scheme would help with the (unsupervised) identiﬁcation of DS that harbor widely different time scales.
Revised: While it is encouraging that the rPLRNN may perform even better than several previous approaches to the vanishing vs. exploding gradients problem, our major goal here was to examine whether our regularization scheme would help with the (unsupervised) identiﬁcation of DS that harbor widely different time scales.

Original: To test this, we used a biophysical, bursting cortical neuron model with one voltage ( V ) and two conductance recovery variables (see Durstewitz (2009)), one slow ( h ) and one fast ( n ; Suppl.
Revised: To test this, we used a biophysical, bursting cortical neuron model with one voltage ( V ) and two conductance recovery variables (see Durstewitz (2009)), one slow ( h ) and one fast ( n ; Suppl.

Original: Reproduction of this DS is challenging since it produces very fast spikes on top of a slow nonlinear oscillation (Fig.
Revised: Reproduction of this DS is challenging since it produces very fast spikes on top of a slow nonlinear oscillation (Fig.

Original: 3 D ).
Revised: 3 D ).

Original: Only short time series (as in scientiﬁc data) of length
Revised: Only short time series (as in scientiﬁc data) of length

Original: T = 1500 from this model were provided for training.
Revised: T = 1500 from this model were provided for training.

Original: rPLRNNs with M = { 8 . . .
Revised: rPLRNNs with M = { 8 . . .

Original: 18 } states were trained, with the regularization factor varied within τ ∈ { 0 , 10 1 , 10 2 , 10 3 , 10 4 , 10 5 } /T .
Revised: 18 } states were trained, with the regularization factor varied within τ ∈ { 0 , 10 1 , 10 2 , 10 3 , 10 4 , 10 5 } /T .

Original: Note that for τ = 0 (no regularization), the approach reduces to the standard PLRNN (Koppe et al., 2019).
Revised: Note that for τ = 0 (no regularization), the approach reduces to the standard PLRNN (Koppe et al., 2019).

Original: Fig.
Revised: Fig.

Original: 3 A conﬁrms our intuition that stronger regularization leads to better DS reconstruction as assessed by the KL divergence between true and generated state distributions (similar results were obtained with ahead-prediction errors as a metric, Fig.
Revised: 3 A conﬁrms our intuition that stronger regularization leads to better DS reconstruction as assessed by the KL divergence between true and generated state distributions (similar results were obtained with ahead-prediction errors as a metric, Fig.

Original: S4 A ), accompanied by a likewise decrease in the MSE between the power spectra of true (suppl.
Revised: S4 A ), accompanied by a likewise decrease in the MSE between the power spectra of true (suppl.

Original: eq. 56) and generated (rPLRNN) voltage traces (Fig.
Revised: eq. 56) and generated (rPLRNN) voltage traces (Fig.

Original: 3 B ).
Revised: 3 B ).

Original: Fig.
Revised: Fig.

Original: 3 D gives an example of voltage traces ( V ) and the slower of the two gating variables ( h ; see Fig.
Revised: 3 D gives an example of voltage traces ( V ) and the slower of the two gating variables ( h ; see Fig.

Original: S5 A for variable n ) freely simulated (i.e., sampled) from the autonomously running rPLRNN.
Revised: S5 A for variable n ) freely simulated (i.e., sampled) from the autonomously running rPLRNN.

Original: This illustrates that our model is in principle capable of capturing both the stiff spike dynamics and the slower oscillations in the second gating variable at the same time.
Revised: This illustrates that our model is in principle capable of capturing both the stiff spike dynamics and the slower oscillations in the second gating variable at the same time.

Original: Fig.
Revised: Fig.

Original: 3 C provides more insight into how the regularization worked: While the high frequency components ( > 50 Hz ) related to the repetitive spiking activity hardly beneﬁted from increasing τ , there was a strong reduction in the MSE computed on the power spectrum for the lower frequency range ( ≤ 50 Hz ), suggesting that increased regularization helps to map slowly evolving components of the dynamics.
Revised: 3 C provides more insight into how the regularization worked: While the high frequency components ( > 50 Hz ) related to the repetitive spiking activity hardly beneﬁted from increasing τ , there was a strong reduction in the MSE computed on the power spectrum for the lower frequency range ( ≤ 50 Hz ), suggesting that increased regularization helps to map slowly evolving components of the dynamics.

Original: This result is more general as shown in Fig.
Revised: This result is more general as shown in Fig.

Original: S6 for another DS example.
Revised: S6 for another DS example.

Original: In contrast, an orthogonality (Vorontsov et al., 2017) or plain L2 constraint on weight matrices did not help at all on this problem (Fig.
Revised: In contrast, an orthogonality (Vorontsov et al., 2017) or plain L2 constraint on weight matrices did not help at all on this problem (Fig.

Original: S4 B ).
Revised: S4 B ).

Original: Further insight into the dynamical mechanisms by which the rPLRNN solves the problem can be obtained by examining the latent dynamics: As shown in Fig.
Revised: Further insight into the dynamical mechanisms by which the rPLRNN solves the problem can be obtained by examining the latent dynamics: As shown in Fig.

Original: 3 E (see also Fig.
Revised: 3 E (see also Fig.

Original: S5), regularized states indeed help to map the slow components of the dynamics, while non-regularized states focus on the fast spikes.
Revised: S5), regularized states indeed help to map the slow components of the dynamics, while non-regularized states focus on the fast spikes.

Original: These observations further corroborate the ﬁndings in Fig.
Revised: These observations further corroborate the ﬁndings in Fig.

Original: 3 C and Fig.
Revised: 3 C and Fig.

Original: S6 C .
Revised: S6 C .

Original: 4. R EGULARIZATION PROPERTIES AND MANIFOLD ATTRACTORS
Revised: 4. R EGULARIZATION PROPERTIES AND MANIFOLD ATTRACTORS

Original: In Figs.
Revised: In Figs.

Original: 2 and 3 we demonstrated that the rPLRNN is able to solve problems and reconstruct dynamics that involve long-range dependencies.
Revised: 2 and 3 we demonstrated that the rPLRNN is able to solve problems and reconstruct dynamics that involve long-range dependencies.

Original: Figs.
Revised: Figs.

Original: 3 A , B furthermore directly conﬁrm that solutions improve with stronger regularization, while Figs.
Revised: 3 A , B furthermore directly conﬁrm that solutions improve with stronger regularization, while Figs.

Original: 3 C , E give insight into the mechanism by which the regularization works.
Revised: 3 C , E give insight into the mechanism by which the regularization works.

Original: To further verify empirically that our speciﬁc form of regularization, eq.
Revised: To further verify empirically that our speciﬁc form of regularization, eq.

Original: 3, is important, Fig.
Revised: 3, is important, Fig.

Original: 2 also shows results for a PLRNN with standard L2 norm on a fraction of
Revised: 2 also shows results for a PLRNN with standard L2 norm on a fraction of

Original: M reg /M
Revised: M reg /M

Original: =
Revised: =

Original: 5 states (L2pPLRNN).
Revised: 5 states (L2pPLRNN).

Original: Fig.
Revised: Fig.

Original: S7 provides additional results for PLRNNs with L2 norm on all weights and for vanilla L2-regularized RNNs.
Revised: S7 provides additional results for PLRNNs with L2 norm on all weights and for vanilla L2-regularized RNNs.

Original: All these systems fell far behind the performance of the rPLRNN on all tasks tested.
Revised: All these systems fell far behind the performance of the rPLRNN on all tasks tested.

Original: Moreover, Fig.
Revised: Moreover, Fig.

Original: 4 reveals that the speciﬁc regularization proposed indeed encourages manifold attractors, and that this is not achieved by a standard L2 regularization:
Revised: 4 reveals that the speciﬁc regularization proposed indeed encourages manifold attractors, and that this is not achieved by a standard L2 regularization:

Original: In contrast to L2PLRNN, as the regularization factor τ is increased, more and more of the maximum absolute eigenvalues around the system’s ﬁxed points (computed according to eq. 8, sect.
Revised: In contrast to L2PLRNN, as the regularization factor τ is increased, more and more of the maximum absolute eigenvalues around the system’s ﬁxed points (computed according to eq. 8, sect.

Original: 6.1.2) cluster on or near 1, indicating directions of marginal stability in state space.
Revised: 6.1.2) cluster on or near 1, indicating directions of marginal stability in state space.

Original: Also, the deviations from 1 become smaller for strongly regularized PLRNNs (Fig.
Revised: Also, the deviations from 1 become smaller for strongly regularized PLRNNs (Fig.

Original: 4 B,D ), indicating a higher precision in attractor tuning.
Revised: 4 B,D ), indicating a higher precision in attractor tuning.

Original: Fig.
Revised: Fig.

Original: S9 in addition conﬁrms that rPLRNN parameters are increasingly driven toward values that would support manifold attractors with stronger regularization.
Revised: S9 in addition conﬁrms that rPLRNN parameters are increasingly driven toward values that would support manifold attractors with stronger regularization.

Original: Fig.
Revised: Fig.

Original: 3 E furthermore suggests that both regularized and non-regularized states are utilized to map the full dynamics.
Revised: 3 E furthermore suggests that both regularized and non-regularized states are utilized to map the full dynamics.

Original: But how should the ratio M reg /M be chosen in practice?
Revised: But how should the ratio M reg /M be chosen in practice?

Original: While for the problems here this meta-parameter was determined through ‘classical’ grid-search and cross-validation, Figs.
Revised: While for the problems here this meta-parameter was determined through ‘classical’ grid-search and cross-validation, Figs.

Original: S
Revised: S

Original: C – E suggest that the precise setting of M reg /M is actually not overly important: Nearly optimal performance is achieved for a broader range M reg /M ∈ [0 . 3 , 0 . 6] on all problems tested.
Revised: C – E suggest that the precise setting of M reg /M is actually not overly important: Nearly optimal performance is achieved for a broader range M reg /M ∈ [0 . 3 , 0 . 6] on all problems tested.

Original: Hence, in practice, setting M reg /M
Revised: Hence, in practice, setting M reg /M

Original: = 0 .
Revised: = 0 .

Original: 5 should mostly work ﬁne.
Revised: 5 should mostly work ﬁne.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work we introduced a simple solution to the long short-term memory problem in RNNs that retains the simplicity and tractability of PLRNNs, yet does not curtail their universal computational capabilities (Koiran et al., 1994; Siegelmann & Sontag, 1995) and their ability to approximate arbitrary DS (Funahashi & Nakamura, 1993; Kimura & Nakano, 1998; Trischler & D’Eleuterio, 2016).
Revised: In this work we introduced a simple solution to the long short-term memory problem in RNNs that retains the simplicity and tractability of PLRNNs, yet does not curtail their universal computational capabilities (Koiran et al., 1994; Siegelmann & Sontag, 1995) and their ability to approximate arbitrary DS (Funahashi & Nakamura, 1993; Kimura & Nakano, 1998; Trischler & D’Eleuterio, 2016).

Original: We achieved this by adding regularization terms to the loss function that encourage the system to form a ‘memory subspace’ (Seung, 1996; Durstewitz, 2003) which would store arbitrary values for, if unperturbed, arbitrarily long periods.
Revised: We achieved this by adding regularization terms to the loss function that encourage the system to form a ‘memory subspace’ (Seung, 1996; Durstewitz, 2003) which would store arbitrary values for, if unperturbed, arbitrarily long periods.

Original: At the same time we did not rigorously enforce this constraint, which allowed the system to capture slow time scales by slightly departing from a perfect manifold attractor.
Revised: At the same time we did not rigorously enforce this constraint, which allowed the system to capture slow time scales by slightly departing from a perfect manifold attractor.

Original: In neuroscience, this has been discussed as a dynamical mechanism for regulating the speed of ﬂow in DS and learning of arbitrary time constants not naturally included qua
Revised: In neuroscience, this has been discussed as a dynamical mechanism for regulating the speed of ﬂow in DS and learning of arbitrary time constants not naturally included qua

Original: RNN design (Durstewitz, 2003; 2004) (Fig.
Revised: RNN design (Durstewitz, 2003; 2004) (Fig.

Original: 1 B ).
Revised: 1 B ).

Original: While other RNN architectures, including vanilla
Revised: While other RNN architectures, including vanilla

Original: RNNs, can, in principle, also develop line attractors to solve speciﬁc tasks (Maheswaranathan et al., 2019), they are generally much harder to train to achieve this and may exhibit less precise attractor tuning (cf.
Revised: RNNs, can, in principle, also develop line attractors to solve speciﬁc tasks (Maheswaranathan et al., 2019), they are generally much harder to train to achieve this and may exhibit less precise attractor tuning (cf.

Original: Fig. 4), which is needed to bridge long time scales (Durstewitz, 2003).
Revised: Fig. 4), which is needed to bridge long time scales (Durstewitz, 2003).

Original: Moreover, part of the PLRNN’s latent space was not regularized at all, leaving the system enough degrees of freedom for realizing arbitrary computations or dynamics (see also Fig.
Revised: Moreover, part of the PLRNN’s latent space was not regularized at all, leaving the system enough degrees of freedom for realizing arbitrary computations or dynamics (see also Fig.

Original: S11 for a chaotic example).
Revised: S11 for a chaotic example).

Original: We showed that the rPLRNN is en par with or outperforms initialization-based approaches, orthogonal
Revised: We showed that the rPLRNN is en par with or outperforms initialization-based approaches, orthogonal

Original: RNNs, and LSTMs on a number of classical benchmarks.
Revised: RNNs, and LSTMs on a number of classical benchmarks.

Original: More importantly, however, the regularization strongly facilitates the identiﬁcation of challenging DS with widely different time scales in
Revised: More importantly, however, the regularization strongly facilitates the identiﬁcation of challenging DS with widely different time scales in

Original: PLRNN-based algorithms for DS reconstruction.
Revised: PLRNN-based algorithms for DS reconstruction.

Original: Similar regularization schemes as proposed here (eq. 3) may, in principle, also be designed for other architectures, but the convenient mathematical form of the PLRNN makes their implementation particularly powerful and straightforward. 
Revised: Similar regularization schemes as proposed here (eq. 3) may, in principle, also be designed for other architectures, but the convenient mathematical form of the PLRNN makes their implementation particularly powerful and straightforward. A CKNOWLEDGEMENTS

Revised: This work was funded by grants from the German Research Foundation (DFG) to DD (Du 354/10-1,

Revised: Du 354/8-2 within SPP 1665) and to GK (TRR265: A06 & B08), and under Germany’s Excellence

Revised: Strategy – EXC-2181 – 390900948 (’Structures’).

Original:  A PPENDIX 6. S UPPLEMENTARY TEXT 6.1. Simple exact PLRNN solution for addition problem
Revised:  A PPENDIX 6. S UPPLEMENTARY TEXT 6.1. Simple exact PLRNN solution for addition problem

Original: The exact PLRNN parameter settings (cf. eq. 1, eq. 2) for solving the addition problem with 2 units (cf.
Revised: The exact PLRNN parameter settings (cf. eq. 1, eq. 2) for solving the addition problem with 2 units (cf.

Original: Fig.
Revised: Fig.

Original: 1 C ) are as follows:
Revised: 1 C ) are as follows:

Original: [Equation]
Revised: [Equation]

Original: 6.1. Computation of ﬁxed points and cycles in PLRNN
Revised: 6.1. Computation of ﬁxed points and cycles in PLRNN

Original: Consider the PLRNN in the form of eq. 4.
Revised: Consider the PLRNN in the form of eq. 4.

Original: For clarity, let us deﬁne d Ω( t ) :
Revised: For clarity, let us deﬁne d Ω( t ) :

Original: = ( d 1 , d 2 , · · · , d M ) as an indicator vector with d m ( z m,t ) :
Revised: = ( d 1 , d 2 , · · · , d M ) as an indicator vector with d m ( z m,t ) :

Original: = d m = 1 for all states z m,t > 0 and zeros otherwise, and
Revised: = d m = 1 for all states z m,t > 0 and zeros otherwise, and

Original: D Ω( t ) :
Revised: D Ω( t ) :

Original: = diag( d Ω( t ) ) as the diagonal matrix formed from this vector.
Revised: = diag( d Ω( t ) ) as the diagonal matrix formed from this vector.

Original: Note that there are at most 2 M distinct matrices W Ω( t ) as deﬁned in eq. 4, depending on the sign of the components of z t .
Revised: Note that there are at most 2 M distinct matrices W Ω( t ) as deﬁned in eq. 4, depending on the sign of the components of z t .

Original: If h = 0 and W Ω( t ) is the identity matrix, then the map F becomes the identity map and so every point z will be a ﬁxed point of F .
Revised: If h = 0 and W Ω( t ) is the identity matrix, then the map F becomes the identity map and so every point z will be a ﬁxed point of F .

Original: Otherwise, the ﬁxed points of F can be found solving the equation
Revised: Otherwise, the ﬁxed points of F can be found solving the equation

Original: F ( z ∗ 1 ) =
Revised: F ( z ∗ 1 ) =

Original: z ∗ 1 as
Revised: z ∗ 1 as

Original: [Equation]
Revised: [Equation]

Original: (8) where z ∗ 1 =
Revised: (8) where z ∗ 1 =

Original: z t ∗ 1 =
Revised: z t ∗ 1 =

Original: z t ∗ 1 − 1 , if det( I − W Ω( t ∗ 1 ) ) =
Revised: z t ∗ 1 − 1 , if det( I − W Ω( t ∗ 1 ) ) =

Original: P W Ω( t ∗ 1) (1) (cid:54) = 0 , i.e. W Ω( t ∗ 1 ) has no eigenvalue equal to 1 .
Revised: P W Ω( t ∗ 1) (1) (cid:54) = 0 , i.e. W Ω( t ∗ 1 ) has no eigenvalue equal to 1 .

Original: Stability and type of ﬁxed points (node, saddle, spiral) can then be determined from the eigenvalues of the Jacobian A + W D Ω( t ∗ 1 ) =
Revised: Stability and type of ﬁxed points (node, saddle, spiral) can then be determined from the eigenvalues of the Jacobian A + W D Ω( t ∗ 1 ) =

Original: W Ω( t ∗ 1 ) (Strogatz (2015)).
Revised: W Ω( t ∗ 1 ) (Strogatz (2015)).

Original: For k > 1 , solving F k ( z ∗ k ) =
Revised: For k > 1 , solving F k ( z ∗ k ) =

Original: z ∗ k , one can obtain a k -cycle of the map F with the periodic points { z ∗ k , F ( z ∗ k ) , F 2 ( z ∗ k ) , · · · , F k − 1 ( z ∗ k ) } .
Revised: z ∗ k , one can obtain a k -cycle of the map F with the periodic points { z ∗ k , F ( z ∗ k ) , F 2 ( z ∗ k ) , · · · , F k − 1 ( z ∗ k ) } .

Original: For this, we ﬁrst compute F k as follows:
Revised: For this, we ﬁrst compute F k as follows:

Original: [Equation]
Revised: [Equation]

Original: in which (cid:81) 
Revised: in which (cid:81) k

Original: k +1 i =
Revised: + [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Ω
Revised: Ω

Original: [Equation]
Revised: [Equation]

Original: Assuming t +( k − 1) :
Revised: Assuming t +( k − 1) :

Original: = t ∗ k , then the k -cycle is given by the ﬁxed point of the k -times iterated map
Revised: = t ∗ k , then the k -cycle is given by the ﬁxed point of the k -times iterated map

Original: F k as 
Revised: F k as z ∗ k = (cid:18) I − k (cid:89) i = W Ω( t ∗ k − i ) (cid:19) − 1 (cid:20) k

Original: [Equation]
Revised: (cid:88) j = k − j +1 (cid:89) i = W Ω( t ∗ k − i ) + I (cid:21) h

Original:  (10) where z ∗ k =
Revised: = H ∗ k h , (10) where z ∗ k =

Original: z t ∗ k =
Revised: z t ∗ k =

Original: z t ∗ k − k , provided that I − (cid:81)
Revised: z t ∗ k − k , provided that I − (cid:81)

Original: ki
Revised: ki

Original: =1 W Ω( t ∗ k − i ) is invertible.
Revised: =1 W Ω( t ∗ k − i ) is invertible.

Original: That is det 
Revised: That is det (cid:18) I − [Equation] i ) (cid:19) =

Revised: P

Original: [Equation]
Revised: [Equation]

Original: I
Revised: W

Original: [Equation]
Revised: [Equation]

Original: and (cid:81)
Revised: ( t [Equation] k

Original: [Equation]
Revised: [Equation]

Revised: i ) (1) (cid:54) = 0 and (cid:81) ki

Revised: =1 W Ω( t ∗ k − i ) :

Original:  k has no eigenvalue equal to 1 .
Revised: = W Ω ∗ k has no eigenvalue equal to 1 .

Original: As for the ﬁxed points, we can determine stability of the k -cycle from the eigenvalues of the Jacobians (cid:81)
Revised: As for the ﬁxed points, we can determine stability of the k -cycle from the eigenvalues of the Jacobians (cid:81)

Original: k +1 i =2 W Ω ( t ∗ k+( k − i )) .
Revised: ki =1 W Ω( t ∗ k − i ) .

Original: It may also be helpful to spell out the recursions in eq.
Revised: It may also be helpful to spell out the recursions in eq.

Original: 5 and eq. 6 in section 3.3 in a bit more detail.
Revised: 5 and eq. 6 in section 3.3 in a bit more detail.

Original: Analogously to the derivations above, for t = 1 , 2 , . . .
Revised: Analogously to the derivations above, for t = 1 , 2 , . . .

Revised: [Equation]

Original: , T we can recursively compute z 2 , z 3 , . . .
Revised: T we can recursively compute z 2 , z 3 , . . .

Original: , z T ( T ∈ N ) as
Revised: , z T ( T ∈ N ) as

Original: [Equation]
Revised: [Equation]

Original: Likewise
Revised: Likewise

Original: [Equation]
Revised: [Equation]

Original: we can write out the derivatives eq.
Revised: we can write out the derivatives eq.

Original: 6 more explicitly as
Revised: 6 more explicitly as

Original: [Equation]
Revised: [Equation]

Original: (12) where ∂
Revised: 12) where ∂

Original: z 2 ∂w mk =
Revised: z 2 ∂w mk =

Original: ( ∂z 1 , 2 ∂w mk · · · ∂z M, 2 ∂w mk ) with ∂z l, 2 ∂w mk = 

Revised: ( ∂z 1 , 2 ∂w mk · · · ∂z M, 2 ∂w mk ) with ∂z l, 2 ∂w mk =

Original: ∀ l (cid:54) = m and ∂z m, 2 ∂w mk =
Revised: ∀ l (cid:54) = m and ∂z m, 2 ∂w mk =

Original: d k z k, 1 .
Revised: d k z k, 1 .

Original: The derivatives w.r
Revised: The derivatives w.r.t.

Original: [Equation]

Original: t. the elements of A and h can be expanded in a similar way, only that the terms D Ω( t ) z t on the last line of eq.
Revised: the elements of A and h can be expanded in a similar way, only that the terms D Ω( t ) z t on the last line of eq.

Original: 12 need to be replaced by just z t for ∂ z t ∂a
Revised: 12 need to be replaced by just z t for ∂ z t ∂a

Original: mm , and by just a vector of 1 ’s for ∂ z t
Revised: mm , and by just a vector of 1 ’s for ∂ z t

Original: ∂h m (also, in these cases, the indicator matrix will be the diagonal matrix 1 ( m,m ) ).
Revised: ∂h m (also, in these cases, the indicator matrix will be the diagonal matrix 1 ( m,m ) ).

Original: 6.1. Proof of Theorem
Revised: 6.1. Proof of Theorem

Original: To state the proof, let us rewrite the derivatives of the loss function L ( W , A , h )
Revised: To state the proof, let us rewrite the derivatives of the loss function L ( W , A , h )

Original: = (cid:80) Tt =
Revised: = (cid:80) Tt =

Original: 1 L t in the following tensor form:
Revised: 1 L t in the following tensor form:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for which the 3 D tensor
Revised: for which the 3 D tensor

Original: [Equation]
Revised: [Equation]

Original: of dimension M × M × M , consists of all the gradient matrices
Revised: of dimension M × M × M , consists of all the gradient matrices

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: where w i ∗ ∈ R M is a row-vector.
Revised: where w i ∗ ∈ R M is a row-vector.

Original: Now, suppose that { z 1 , z 2 , z 3 , . . .
Revised: Now, suppose that { z 1 , z 2 , z 3 , . . .

Original: } is an orbit of the system which converges to a stable ﬁxed point, i.e. lim T →∞ z T =
Revised: } is an orbit of the system which converges to a stable ﬁxed point, i.e. lim T →∞ z T =

Original: z ∗ k .
Revised: z ∗ k .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original:  and solim
Revised: 16) and so

Original: [Equation]
Revised: [Equation]

Original: Assume that lim T →∞ (cid:0) W Ω( T − 1) (cid:1)
Revised: Assume that lim T →∞ (cid:0) W Ω( T − 1) (cid:1)

Original: =
Revised: =

Original: L .
Revised: L .

Original: Since eq. 17 holds for every z ∗ 1 , then substituting z ∗ 1 =
Revised: Since eq. 17 holds for every z ∗ 1 , then substituting z ∗ 1 =

Original: e T 1 = (1 , 0 , · · · , 0) T in eq.
Revised: e T 1 = (1 , 0 , · · · , 0) T in eq.

Original: 17, we can prove that the ﬁrst column of L equals the ﬁrst column of
Revised: 17, we can prove that the ﬁrst column of L equals the ﬁrst column of

Original: W Ω( t ∗ 1 ) .
Revised: W Ω( t ∗ 1 ) .

Original: Performing the same procedure for z ∗ 1 =
Revised: Performing the same procedure for z ∗ 1 =

Original: e Ti , i = 2 , 3 , · · · , M , yields
Revised: e Ti , i = 2 , 3 , · · · , M , yields

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Also, for every i ∈ N (1 < i < ∞ )
Revised: Also, for every i ∈ N (1 < i < ∞ )

Original: [Equation]
Revised: [Equation]

Original: 19) i.e.
Revised: 19) i.e.

Original: ∀
Revised: ∀

Original: [Equation]
Revised: [Equation]

Revised: >

Revised: [Equation]

Revised: ∈

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: ⇒

Original: [Equation]
Revised: [Equation]

Revised: W

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: ) − W

Original:  Thus, (cid:13)(cid:13) W Ω( T − i ) (cid:13)(cid:13) − (cid:13)(cid:13) W Ω
Revised: [Equation] (cid:13) [Equation] Thus, (cid:13)(cid:13) W Ω( T − i ) (cid:13)(cid:13) − (cid:13)(cid:13) W Ω

Original: ( t ∗ 1 ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) W Ω
Revised: ( t ∗ 1 ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) W Ω

Original: ( T − i )
Revised: ( T − i )

Original: − W Ω
Revised: − W Ω

Original: ( t ∗ 1 ) (cid:13)(cid:13) gives ∀ (cid:15) >
Revised: ( t ∗ 1 ) (cid:13)(cid:13) gives ∀ (cid:15) >

Original: 0 ∃ N ∈ N s.t.
Revised: 0 ∃ N ∈ N s.t.

Original: T − i ≥ N = ⇒ (cid:13)(cid:13) W Ω
Revised: T − i ≥ N = ⇒ (cid:13)(cid:13) W Ω

Original: ( T − i )
Revised: ( T − i )

Original: (cid:13)(cid:13) ≤ (cid:13)(cid:13) W Ω( t ∗ 1 ) (cid:13)(cid:13)
Revised: (cid:13)(cid:13) ≤ (cid:13)(cid:13) W Ω( t ∗ 1 ) (cid:13)(cid:13)

Original: + (cid:15).
Revised: + (cid:15).

Original: Since T − 1 > T − 2 > · · · > T − i ≥ N , so ∀ (cid:15) > 0 (cid:13)(cid:13) W Ω( T − i ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) W Ω( t ∗ 1 ) (cid:13)(cid:13) + (cid:15), i = 1 , 2 , · · · , T − N.Hence ∀ (cid:15) > 0 (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) [Equation] i =
Revised: Since T − 1 > T − 2 > · · · > T − i ≥ N , so ∀ (cid:15) > 0 (cid:13)(cid:13) W Ω( T − i ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) W Ω( t ∗ 1 ) (cid:13)(cid:13) + (cid:15), i = 1 , 2 , · · · , T − N.

Original: [Equation]

Original: W

Original: [Equation] (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)≤ [Equation] (cid:89)
Revised: Hence ∀ (cid:15) > 0 (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)

Original: [Equation]

Original: =

Original: [Equation]

Original: (cid:13)(cid:13) [Equation] [Equation] (cid:13)(cid:13)
Revised: T

Original: [Equation]
Revised: [Equation]

Original: (cid:16) (cid:13)(cid:13) [Equation] [Equation] (cid:13)(cid:13)
Revised: ≤

Original: + (cid:15) (cid:17)

Original: [Equation]
Revised: [Equation]

Original: − N .

Original: If (cid:13)(cid:13) W Ω( t ∗ 1 ) (cid:13)(cid:13) < 1 , then for any (cid:15) < 1 , considering ¯ (cid:15) ≤ (cid:15) + (cid:107) W Ω( t ∗ 1) (cid:107)< 1 , it is concluded that[Equation] T [Equation]
Revised: If (cid:13)(cid:13) W Ω( t ∗ 1 ) (cid:13)(cid:13) < 1 , then for any (cid:15) < 1 , considering ¯ (cid:15) ≤ (cid:15) + (cid:107) W Ω( t ∗ 1) (cid:107)< 1 , it is concluded that

Original: =

Original: [Equation]

Original: (cid:15) (cid:17)

Original: [Equation]
Revised: [Equation]

Original: = 0 .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: If the orbit { z 1 , z 2 , z 3 , . . . } tends to a stable k -cycle ( k > 1) with the periodic points
Revised: If the orbit { z 1 , z 2 , z 3 , . . . } tends to a stable k -cycle ( k > 1) with the periodic points

Original: [Equation]
Revised: [Equation]

Original: then, denoting the stable k -cycle by
Revised: then, denoting the stable k -cycle by

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Hence, there exists a neighborhood U of Γ k and k sub-sequences { z T kn } ∞ n =1 , { z T kn +1 } ∞ n =1 , · · · , { z 
Revised: Hence, there exists a neighborhood U of Γ k and k sub-sequences { z T kn } ∞ n =1 , { z T kn +1 } ∞ n =1 , · · · , { z T kn +

Original: T kn +( k − 1) } ∞ n =1 of the sequence { z T } ∞ T =1 such that these sub-sequences belong to U and
Revised: [Equation] k [Equation] 1) } ∞ n =1 of the sequence { z T } ∞ T =1 such that these sub-sequences belong to U and

Original: [Equation]
Revised: [Equation]

Revised: iii)

Revised: for

Revised: every

Revised: z T ∈

Original: iii) for every z T ∈ U there is some s ∈{ 0 , 1 , 2 , · · · , k −1 } such that z T ∈ { z T kn + s } ∞ n =
Revised: U there is some s ∈ k − such that z T ∈ { z T kn + s } ∞ n =

Original: [Equation]
Revised: [Equation]

Original: In this case, for every z T ∈ U with z T ∈ { z T kn + s } ∞ n =1 we have lim T →∞ z T =
Revised: In this case, for every z T ∈ U with z T ∈ { z T kn + s } ∞ n =1 we have lim T →∞ z T =

Original: z t ∗ k − s for some s = 0 , 1 , 2 , · · · , k − 1 .
Revised: z t ∗ k − s for some s = 0 , 1 , 2 , · · · , k − 1 .

Original: Therefore, continuity of F implies that lim T →∞ F ( z T ) =
Revised: Therefore, continuity of F implies that lim T →∞ F ( z T ) =

Original: F ( z t ∗ k − s ) and so
Revised: F ( z t ∗ k − s ) and so

Original: [Equation]
Revised: [Equation]

Original: Thus, similarly, we can prove that
Revised: Thus, similarly, we can prove that

Original: [Equation]
Revised: [Equation]

Original: Analogously, for every i ∈ N (1 < i < ∞ )
Revised: Analogously, for every i ∈ N (1 < i < ∞ )

Original: [Equation]
Revised: [Equation]

Original: On the other hand, (cid:13)(cid:13) W Ω( t
Revised: On the other hand, (cid:13)(cid:13) W Ω( t

Original: [Equation]
Revised: [Equation]

Original: k − s i ) (cid:13)(cid:13) < 1 for all s i ∈ { 0 , 1 , 2 , · · · , k − 1 } .
Revised: k − s i [Equation] (cid:13)(cid:13) < 1 for all s i ∈ { 0 , 1 , 2 , · · · , k − 1 } .

Original: So, without loss of generality, assuming
Revised: So, without loss of generality, assuming

Original: [Equation]
Revised: [Equation]

Original: we can again obtain eq.
Revised: we can again obtain eq.

Original: Since { z T − 1 } ∞ T =1 is a convergent sequence, so it is bounded, i.e. there exists a real number q >such that ||
Revised: Since { z T − 1 } ∞ T =1 is a convergent sequence, so it is bounded, i.e. there exists a real number q >such that ||

Original: z T − 1 || ≤ q for all T ∈ N .
Revised: z T − 1 || ≤ q for all T ∈ N .

Original: Furthermore, (cid:13) (cid:13) D Ω
Revised: Furthermore, (cid:13) (cid:13) D Ω

Original: ( T − 1) (cid:13) (cid:13) ≤ 1 for all T .
Revised: ( T − 1) (cid:13) (cid:13) ≤ 1 for all T .

Original: Therefore, by eq.
Revised: Therefore, by eq.

Revised: [Equation]

Revised: [Equation]

Revised: (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Revised: (cid:89)

Original: [Equation]
Revised: [Equation]

Original:  T 
Revised: (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:13) [Equation] ( T − j )

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13) (cid:13)(cid:21)

Original: [Equation]
Revised: [Equation]

Original: T
Revised: (cid:18) 1 + [Equation] (cid:88) [Equation]

Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Revised: ( T

Original: [Equation]
Revised: [Equation]

Revised: i ) (cid:13)(cid:13) (cid:19) .

Original: On the other hand, lim T →∞
Revised: On the other hand, lim T →∞

Original: [Equation]
Revised: [Equation]

Original: is convergent by the root test, since lim
Revised: is convergent by the root test, since lim

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original:  i.e., by eq.
Revised: 35) i.e., by eq.

Original: 14 and eq.
Revised: 14 and eq.

Original: 15, the 2 -norm of total gradient matrices and hence (cid:13)(cid:13)
Revised: 15, the 2 -norm of total gradient matrices and hence (cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: z t ∂ W (cid:13)(cid:13) 2 will not diverge (explode) under the assumptions of Theorem 1.
Revised: z t ∂ W (cid:13)(cid:13) 2 will not diverge (explode) under the assumptions of Theorem 1.

Original: Analogously, we can prove that (cid:13)(cid:13) ∂
Revised: Analogously, we can prove that (cid:13)(cid:13) ∂

Original: z T ∂ A (cid:13)(cid:13) 2 and (cid:13)(cid:13) ∂
Revised: z T ∂ A (cid:13)(cid:13) 2 and (cid:13)(cid:13) ∂

Original: z T ∂ h (cid:13)(cid:13) 2 will not diverge either.
Revised: z T ∂ h (cid:13)(cid:13) 2 will not diverge either.

Original: Since, similar as in the derivations above, it can be shown that relation eq.
Revised: Since, similar as in the derivations above, it can be shown that relation eq.

Original: 35 is true for (cid:13) (cid:13) (cid:13) ∂
Revised: 35 is true for (cid:13) (cid:13) (cid:13) ∂

Original: z T ∂a mm (cid:13) (cid:13) (cid:13) with q = ¯ q , where ¯ q is the upper bound of (cid:107) z T (cid:107) , as { z T } ∞ T =1 is convergent.
Revised: z T ∂a mm (cid:13) (cid:13) (cid:13) with q = ¯ q , where ¯ q is the upper bound of (cid:107) z T (cid:107) , as { z T } ∞ T =1 is convergent.

Original: Furthermore, relation eq.
Revised: Furthermore, relation eq.

Original: 35 also holds for (cid:13)(cid:13)(cid:13) ∂
Revised: 35 also holds for (cid:13)(cid:13)(cid:13) ∂

Original: z T ∂h m (cid:13)(cid:13)(cid:13) with q = 1 .
Revised: z T ∂h m (cid:13)(cid:13)(cid:13) with q = 1 .

Original: Remark 2.1.
Revised: Remark 2.1.

Original: By eq.
Revised: By eq.

Original: 24 the Jacobian parts (cid:13)(cid:13)(cid:13)
Revised: 24 the Jacobian parts (cid:13)(cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: z T ∂ z t (cid:13)(cid:13)(cid:13) 2 connecting any two states z T and z t , T > t , will not diverge either.
Revised: z T ∂ z t (cid:13)(cid:13)(cid:13) 2 connecting any two states z T and z t , T > t , will not diverge either.

Original: Corollary 2.1.
Revised: Corollary 2.1.

Original: The results of Theorem 1 are also true if W Ω( t ∗ k ) is a normal matrix with no eigenvalue equal to one.
Revised: The results of Theorem 1 are also true if W Ω( t ∗ k ) is a normal matrix with no eigenvalue equal to one.

Original: Proof.
Revised: Proof.

Original: If W Ω( t ∗ k ) is normal, then (cid:13)(cid:13) W Ω( t ∗ k ) (cid:13)(cid:13)
Revised: If W Ω( t ∗ k ) is normal, then (cid:13)(cid:13) W Ω( t ∗ k ) (cid:13)(cid:13)

Original: =
Revised: =

Original: ρ ( W Ω( t ∗ k ) ) < 1 which satisﬁes the conditions of
Revised: ρ ( W Ω( t ∗ k ) ) < 1 which satisﬁes the conditions of

Original: Theorem 1.
Revised: Theorem 1.

Original: 6.1. Proof of Theorem
Revised: 6.1. Proof of Theorem

Original: Let A , W and D Ω( k ) , t < k ≤ T , be partitioned as follows
Revised: Let A , W and D Ω( k ) , t < k ≤ T , be partitioned as follows

Original: A =(cid:32)
Revised: A =

Original: [Equation]
Revised: [Equation]

Original: (36) where I M reg × M reg :
Revised: (36) where I M reg × M reg :

Original: = I reg ∈ R M reg
Revised: = I reg ∈ R M reg

Original: × M reg , O M reg × M reg :
Revised: × M reg , O M reg × M reg :

Original: = O reg ∈ R M reg × M reg , O , S ∈
Revised: = O reg ∈ R M reg × M reg , O , S ∈

Original: R ( M − M reg )
Revised: R ( M − M reg )

Original: × M reg , A { M reg +1: M,M reg +1: M } :=
Revised: × M reg , A { M reg +1: M,M reg +1: M } :=

Original: A nreg ∈ R ( M − M reg ) ×
Revised: A nreg ∈ R ( M − M reg ) ×

Original: ( M − M reg ) is a diagonal submatrix, W { M reg +1: M,M reg +1: M } := W nreg ∈ R ( M − M reg ) ×
Revised: ( M − M reg ) is a diagonal submatrix, W { M reg +1: M,M reg +1: M } := W nreg ∈ R ( M − M reg ) ×

Original: ( M − M reg ) is an off-diagonal sub-matrix (cf.
Revised: ( M − M reg ) is an off-diagonal sub-matrix (cf.

Original: Fig.
Revised: Fig.

Original: S1).
Revised: S1).

Original: Moreover, D kM reg × M reg := D kreg ∈ R M reg × M reg and D k { M reg +1:
Revised: Moreover, D kM reg × M reg := D kreg ∈ R M reg × M reg and D k { M reg +1:

Original: M,M reg +1: M } :=
Revised: M,M reg +1: M } :=

Original: D knreg ∈ R
Revised: D knreg ∈ R

Original: ( M − M reg ) ×
Revised: ( M − M reg ) ×

Original: ( M − M reg ) are diagonal sub-matrices.
Revised: ( M − M reg ) are diagonal sub-matrices.

Original: Then, we have (cid:89) t<
Revised: Then, we have (cid:89) t<k ≤ T

Original: [Equation]
Revised: W Ω( k ) =

Original: k ) = (cid:89) t<k [Equation] (cid:32)
Revised: (cid:89) t<k ≤ T (cid:32)

Original: I reg
Revised: I reg

Original: O T
Revised: O T

Original: S D kreg
Revised: S D kreg

Original: A

Original: [Equation] + W [Equation] (cid:33) :
Revised: A nreg + W nreg D knreg (cid:33) :

Original: = (cid:89) t<k ≤ T (cid:32)
Revised: = (cid:89) t<k ≤ T (cid:32)

Original: I reg
Revised: I reg

Original: O T
Revised: O T

Original: S D kreg
Revised: S D kreg

Original: W knreg (cid:33) =
Revised: W knreg (cid:33) =

Original: (cid:32)
Revised: (cid:32)

Original: I reg
Revised: I reg

Original: O T
Revised: O T

Revised: S D

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Original: (cid:81)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: <
Revised: (cid:0) (cid:81) [Equation] <k ≤ t + j − [Equation]

Revised: W k nreg (cid:1)

Original: [Equation]
Revised: [Equation]

Revised: D t

Revised: [Equation]

Revised: j reg (cid:81) t<k ≤ T W k nreg .

Original: (cid:33) (37)
Revised: (cid:33) (37)

Original: Therefore, considering the 2-norm, we obtain
Revised: Therefore, considering the 2-norm, we obtain

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: 39) where W T − t := (cid:81) t<k ≤ T W knreg .
Revised: [Equation] where W T − t := (cid:81) t<k ≤ T W knreg .

Original: Therefore, eq. 38 and eq.
Revised: Therefore, eq. 38 and eq.

Original: 39 yield
Revised: 39 yield

Revised: [Equation]

Revised: low ≤

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we assumed that the non-regularized subsystem ( z M reg +1 . . .
Revised: Furthermore, we assumed that the non-regularized subsystem ( z M reg +1 . . .

Original: z M ) , if considered in isolation, satisﬁes Theorem 1.
Revised: z M ) , if considered in isolation, satisﬁes Theorem 1.

Original: Hence, similar to the proof of Theorem 1, it is concluded that
Revised: Hence, similar to the proof of Theorem 1, it is concluded that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, by deﬁnition of D Ω( k ) , for every t < k ≤ T , we have (cid:13)(cid:13) D kreg (cid:13)(cid:13) ≤ 1 and so
Revised: On the other hand, by deﬁnition of D Ω( k ) , for every t < k ≤ T , we have (cid:13)(cid:13) D kreg (cid:13)(cid:13) ≤ 1 and so

Original: [Equation]
Revised: [Equation]

Original: 41) which, in accordance with the the assumptions of Theorem 1, by convergence of

Original: [Equation]
Revised: [Equation]

Original: implies
Revised: which, in accordance with the the assumptions of Theorem 1, by convergence of

Original: [Equation]
Revised: [Equation]

Original: t +
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:107) S

Original: [Equation]
Revised: [Equation]

Original: M nreg [Equation] [Equation] Thus, denoting Q := S D t +1 reg + (cid:80)
Revised: Thus, denoting Q := S D t +1 reg + (cid:80)

Original: Tj =2 (cid:0) (cid:81) t<k ≤ t + j − 1 W knreg S D t + j reg (cid:19) , from eq.
Revised: Tj =2 (cid:0) (cid:81) t<k ≤ t + j − 1 W knreg S D t + j reg (cid:19) , from eq.

Original: 42 we deduce that
Revised: 42 we deduce that

Original: [Equation]
Revised: [Equation]

Original: Now, if T − t tends to ∞ , then eq. 38, eq. 40 and eq.
Revised: Now, if T − t tends to ∞ , then eq. 38, eq. 40 and eq.

Original: 43 result in
Revised: 43 result in

Original: [Equation]
Revised: [Equation]

Original: Remark 2.2.
Revised: Remark 2.2.

Original: If (cid:107) S (cid:107) = 0 , then (cid:13) (cid:13) (cid:13) ∂
Revised: If (cid:107) S (cid:107) = 0 , then (cid:13) (cid:13) (cid:13) ∂

Original: z T ∂ z t (cid:13) (cid:13) (cid:13) → 1 as T − t → ∞ .
Revised: z T ∂ z t (cid:13) (cid:13) (cid:13) → 1 as T − t → ∞ .

Original: 6.1. Details on EM algorithm and DS reconstruction
Revised: 6.1. Details on EM algorithm and DS reconstruction

Original: For DS reconstruction we request that the latent RNN approximates the true generating system of equations, which is a taller order than learning the mapping S → X or predicting future values in a time series (cf. sect. 3.5).
Revised: For DS reconstruction we request that the latent RNN approximates the true generating system of equations, which is a taller order than learning the mapping S → X or predicting future values in a time series (cf. sect. 3.5).

Original: This point has important implications for the design of models, inference algorithms and performance metrics if the primary goal is DS reconstruction rather than ‘mere’ time series forecasting.
Revised: This point has important implications for the design of models, inference algorithms and performance metrics if the primary goal is DS reconstruction rather than ‘mere’ time series forecasting.

Original: In this context we consider the fully probabilistic, generative RNN eq.
Revised: In this context we consider the fully probabilistic, generative RNN eq.

Original: Together with eq.
Revised: Together with eq.

Original: 2 (where we take g ( z t ) =
Revised: 2 (where we take g ( z t ) =

Original: φ ( z t ))
Revised: φ ( z t ))

Original: this gives the typical form of a nonlinear state space model (Durbin & Koopman, 2012) with observation and process noise.
Revised: this gives the typical form of a nonlinear state space model (Durbin & Koopman, 2012) with observation and process noise.

Original: We solve for the parameters θ = { A , W , C , h , µ 0 , Σ , B , Γ } by maximum likelihood, for which an efﬁcient
Revised: We solve for the parameters θ = { A , W , C , h , µ 0 , Σ , B , Γ } by maximum likelihood, for which an efﬁcient

Original: Expectation-Maximization (EM) algorithm has recently been suggested (Durstewitz, 2017; Koppe et al., 2019), which we will summarize here.
Revised: Expectation-Maximization (EM) algorithm has recently been suggested (Durstewitz, 2017; Koppe et al., 2019), which we will summarize here.

Original: Since the involved integrals are not tractable, we start off from the evidence-lower bound (ELBO) to the log-likelihood which can be rewritten in various useful ways:
Revised: Since the involved integrals are not tractable, we start off from the evidence-lower bound (ELBO) to the log-likelihood which can be rewritten in various useful ways:

Original: [Equation]
Revised: [Equation]

Original: In the E-step, given a current estimate θ ∗ for the parameters, we seek to determine the posterior p θ ( Z | X ) which we approximate by a global Gaussian q ( Z | X ) instantiated by the maximizer (mode)
Revised: In the E-step, given a current estimate θ ∗ for the parameters, we seek to determine the posterior p θ ( Z | X ) which we approximate by a global Gaussian q ( Z | X ) instantiated by the maximizer (mode)

Original: Z ∗ of p θ ( Z | X ) as an estimator of the mean, and the negative inverse Hessian around this maximizer as an estimator of the state covariance, i.e.
Revised: Z ∗ of p θ ( Z | X ) as an estimator of the mean, and the negative inverse Hessian around this maximizer as an estimator of the state covariance, i.e.

Original: [Equation]
Revised: [Equation]

Original: )] , (46) since Z integrates out in p θ ( X ) (equivalently, this result can be derived from a Laplace approximation to the log-likelihood, log p ( X | θ )
Revised: ] , (46) since Z integrates out in p θ ( X ) (equivalently, this result can be derived from a Laplace approximation to the log-likelihood, log p ( X | θ )

Original: ≈ log p θ ( X | Z ∗ )
Revised: ≈ log p θ ( X | Z ∗ )

Original: +log p θ ( Z ∗ ) − 12 log |−
Revised: +log p θ ( Z ∗ ) − 12 log |−

Original: L ∗ | +const , where
Revised: L ∗ | +const , where

Original: L ∗ is the Hessian evaluated at the maximizer).
Revised: L ∗ is the Hessian evaluated at the maximizer).

Original: We solve this optimization problem by a ﬁxed-point iteration scheme that efﬁciently exploits the model’s piecewise linear structure, as detailed below.
Revised: We solve this optimization problem by a ﬁxed-point iteration scheme that efﬁciently exploits the model’s piecewise linear structure, as detailed below.

Original: Using this approximate posterior for p θ ( Z | X ) , based on the model’s piecewise-linear structure most of the expectation values E z ∼ q
Revised: Using this approximate posterior for p θ ( Z | X ) , based on the model’s piecewise-linear structure most of the expectation values E z ∼ q

Original: [ φ ( z )] , E z ∼ q (cid:2) φ ( z ) z T (cid:3) , and E z ∼ q (cid:2) φ ( z ) φ ( z ) T (cid:3) , could be solved for (semi-)analytically (where z is the concatenated vector form of Z , as in Suppl. 6.1.5).
Revised: [ φ ( z )] , E z ∼ q (cid:2) φ ( z ) z T (cid:3) , and E z ∼ q (cid:2) φ ( z ) φ ( z ) T (cid:3) , could be solved for (semi-)analytically (where z is the concatenated vector form of Z , as in Suppl. 6.1.5).

Original: In the Mstep, we seek θ ∗ := arg max θ L ( θ , q ∗ ) , assuming proposal density q ∗ to be given from the E-step, which for a Gaussian observation model amounts to a simple linear regression problem (see Suppl. eq. 50).
Revised: In the Mstep, we seek θ ∗ := arg max θ L ( θ , q ∗ ) , assuming proposal density q ∗ to be given from the E-step, which for a Gaussian observation model amounts to a simple linear regression problem (see Suppl. eq. 50).

Original: To force the PLRNN to really capture the underlying DS in its governing equations, we use a previously suggested (Koppe et al., 2019) stepwise annealing protocol that gradually shifts the burden of ﬁtting the observations X from the observation model eq.
Revised: To force the PLRNN to really capture the underlying DS in its governing equations, we use a previously suggested (Koppe et al., 2019) stepwise annealing protocol that gradually shifts the burden of ﬁtting the observations X from the observation model eq.

Original: 2 to the latent RNN model eq. 

Revised: 2 to the latent RNN model eq.

Original: during training, the idea of which is to establish a mapping from latent states Z to observations
Revised: during training, the idea of which is to establish a mapping from latent states Z to observations

Original: X ﬁrst, ﬁxing this, and then enforcing the temporal consistency constraints implied by eq. 1 while accounting for the actual observations.
Revised: X ﬁrst, ﬁxing this, and then enforcing the temporal consistency constraints implied by eq. 1 while accounting for the actual observations.

Original: Now we brieﬂy outline the ﬁxed-point-iteration algorithm for solving the maximization problem in eq.
Revised: Now we brieﬂy outline the ﬁxed-point-iteration algorithm for solving the maximization problem in eq.

Original: 46 (for more details see Durstewitz (2017); Koppe et al.
Revised: 46 (for more details see Durstewitz (2017); Koppe et al.

Original: Given a Gaussian latent
Revised: Given a Gaussian latent

Original: PLRNN and a Gaussian observation model, the joint density p ( X , Z ) will be piecewise Gaussian, hence eq.
Revised: PLRNN and a Gaussian observation model, the joint density p ( X , Z ) will be piecewise Gaussian, hence eq.

Original: 46 piecewise quadratic in Z .
Revised: 46 piecewise quadratic in Z .

Original: Let us concatenate all state variables across m and t into one long column vector z
Revised: Let us concatenate all state variables across m and t into one long column vector z

Original: = ( z 1 , 1 , . . .
Revised: = ( z 1 , 1 , . . .

Original: , z M, 1 , . . .
Revised: , z M, 1 , . . .

Original: , z 1 ,T , . . .
Revised: , z 1 ,T , . . .

Original: , z M,T ) T , arrange matrices A , W into large MT × MT block tri-diagonal matrices, deﬁne d Ω := (cid:0) 1 z 1 , 1 > 0 , 1 z 2 , 1 > 0 , . .
Revised: , z M,T ) T , arrange matrices A , W into large MT × MT block tri-diagonal matrices, deﬁne d Ω := (cid:0) 1 z 1 , 1 > 0 , 1 z 2 , 1 > 0 , . .

Original: , 1 z M,T > 0 (cid:1) T as an indicator vector with a 1 for all states z m,t > 0 and zeros otherwise, and D Ω := diag( d Ω ) as the diagonal matrix formed from this vector.
Revised: , 1 z M,T > 0 (cid:1) T as an indicator vector with a 1 for all states z m,t > 0 and zeros otherwise, and D Ω := diag( d Ω ) as the diagonal matrix formed from this vector.

Original: Collecting all terms quadratic, linear, or constant in z , we 
Revised: Collecting all terms quadratic, linear, or constant in z , we can then write down the optimization criterion in the form

Original: [Table]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In essence, the algorithm now iterates between the two steps:1.
Revised: In essence, the algorithm now iterates between the two steps:

Original: Given ﬁxed D Ω , solve
Revised: Given ﬁxed D Ω , solve

Original: [Equation]
Revised: [Equation]

Original: Given ﬁxed z ∗ , recompute D Ω
Revised: Given ﬁxed z ∗ , recompute D Ω

Original: until either convergence or one of several stopping criteria (partly likelihood-based, partly to avoid loops) is reached.
Revised: until either convergence or one of several stopping criteria (partly likelihood-based, partly to avoid loops) is reached.

Original: The solution may afterwards be reﬁned by one quadratic programming step.
Revised: The solution may afterwards be reﬁned by one quadratic programming step.

Original: Numerical experiments showed this algorithm to be very fast and efﬁcient (Durstewitz, 2017; Koppe et al., 2019).
Revised: Numerical experiments showed this algorithm to be very fast and efﬁcient (Durstewitz, 2017; Koppe et al., 2019).

Original: At z ∗ , an estimate of the state covariance is then obtained as the inverse negative
Revised: At z ∗ , an estimate of the state covariance is then obtained as the inverse negative

Original: Hessian,
Revised: Hessian,

Original: [Equation]
Revised: [Equation]

Original: In the M-step, using the proposal density q ∗ from the E-step, the solution to the maximization problem θ ∗ := arg max θ
Revised: In the M-step, using the proposal density q ∗ from the E-step, the solution to the maximization problem θ ∗ := arg max θ

Original: L ( θ , q ∗ ) , can generally be expressed in the form
Revised: L ( θ , q ∗ ) , can generally be expressed in the form

Original: [Equation]
Revised: [Equation]

Original: where, for the latent model, eq. 1, α t = z t and β t := (cid:2)
Revised: where, for the latent model, eq. 1, α t = z t and β t := (cid:2)

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: T ∈ R 2 M + K +1 , and for the observation model, eq. 2, α t = x t and β t = g ( z t ) .
Revised: T ∈ R 2 M + K +1 , and for the observation model, eq. 2, α t = x t and β t = g ( z t ) .

Original: 6.1. More details on DS performance measure
Revised: 6.1. More details on DS performance measure

Original: As argued before (Koppe et al., 2019; Wood, 2010), in DS reconstruction we require that the RNN captures the underlying attractor geometries and state space properties.
Revised: As argued before (Koppe et al., 2019; Wood, 2010), in DS reconstruction we require that the RNN captures the underlying attractor geometries and state space properties.

Original: This does not necessarily entail that the reconstructed system could predict future time series observations more than a few time steps ahead, and vice versa.
Revised: This does not necessarily entail that the reconstructed system could predict future time series observations more than a few time steps ahead, and vice versa.

Original: For instance, if the underlying attractor is chaotic, even if we had the exact true system available, with a tiny bit of noise trajectories starting from the same initial condition will quickly diverge and ahead-prediction errors become essentially meaningless as a DS performance metric (Fig. S2 B ).
Revised: For instance, if the underlying attractor is chaotic, even if we had the exact true system available, with a tiny bit of noise trajectories starting from the same initial condition will quickly diverge and ahead-prediction errors become essentially meaningless as a DS performance metric (Fig. S2 B ).

Original: To quantify how well an inferred PLRNN captured the underlying dynamics we therefore followed
Revised: To quantify how well an inferred PLRNN captured the underlying dynamics we therefore followed

Original: Koppe et al.
Revised: Koppe et al.

Original: and used the Kullback-Leibler divergence between the true and reproduced probability distributions across states in state space, thus assessing the agreement in attractor geometries (cf.
Revised: and used the Kullback-Leibler divergence between the true and reproduced probability distributions across states in state space, thus assessing the agreement in attractor geometries (cf.

Original: Takens (1981); Sauer et al.
Revised: Takens (1981); Sauer et al.

Original: (1991)) rather than in precise matching of time series,
Revised: (1991)) rather than in precise matching of time series,

Original: [Equation]
Revised: [Equation]

Original: where p true ( x ) is the true distribution of observations across state space (not time!), p gen ( x | z ) is the distribution of observations generated by running the inferred PLRNN, and the sum indicates a spatial discretization (binning) of the observed state space.
Revised: where p true ( x ) is the true distribution of observations across state space (not time!), p gen ( x | z ) is the distribution of observations generated by running the inferred PLRNN, and the sum indicates a spatial discretization (binning) of the observed state space.

Original: We emphasize that ˆ p ( k ) gen ( x | z ) is obtained from freely simulated trajectories, i.e. drawn from the prior ˆ p ( z ) speciﬁed by eq.
Revised: We emphasize that ˆ p ( k ) gen ( x | z ) is obtained from freely simulated trajectories, i.e. drawn from the prior ˆ p ( z ) speciﬁed by eq.

Original: 1, not from the inferred posteriors ˆ p ( z | x train ) .
Revised: 1, not from the inferred posteriors ˆ p ( z | x train ) .

Original: In addition, to assess reproduction of time scales by the inferred
Revised: In addition, to assess reproduction of time scales by the inferred

Original: PLRNN, the average MSE between the power spectra of the true and generated time series was computed, as displayed in Fig.
Revised: PLRNN, the average MSE between the power spectra of the true and generated time series was computed, as displayed in Fig.

Original: 3 B – C .
Revised: 3 B – C .

Original: The measure D KL introduced above only works for situations where the ground truth p true ( X ) is known.
Revised: The measure D KL introduced above only works for situations where the ground truth p true ( X ) is known.

Original: Following Koppe et al.
Revised: Following Koppe et al.

Original: , we next brieﬂy indicate how a proxy for D KL may be obtained in empirical situations where no ground truth is available.
Revised: , we next brieﬂy indicate how a proxy for D KL may be obtained in empirical situations where no ground truth is available.

Original: Reasoning that for a well reconstructed DS the inferred posterior p inf ( z | x ) given the observations should be a good representative of the prior generative dynamics p gen ( z ) , one may use the Kullback-Leibler divergence between the distribution over latent states, obtained by sampling from the prior density p gen ( z ) , and the (dataconstrained) posterior distribution p inf ( z | x ) (where z ∈ R M × 1 and x ∈ R N × 1 ), taken across the system’s state space:
Revised: Reasoning that for a well reconstructed DS the inferred posterior p inf ( z | x ) given the observations should be a good representative of the prior generative dynamics p gen ( z ) , one may use the Kullback-Leibler divergence between the distribution over latent states, obtained by sampling from the prior density p gen ( z ) , and the (dataconstrained) posterior distribution p inf ( z | x ) (where z ∈ R M × 1 and x ∈ R N × 1 ), taken across the system’s state space:

Original: [Equation]
Revised: [Equation]

Original: As evaluating this integral is difﬁcult, one could further approximate p inf ( z | x ) and p gen ( z ) by
Revised: As evaluating this integral is difﬁcult, one could further approximate p inf ( z | x ) and p gen ( z ) by

Original: Gaussian mixtures across trajectories, i.e.
Revised: Gaussian mixtures across trajectories, i.e.

Original: [Equation]
Revised: [Equation]

Original: and p
Revised: and p

Original: [Equation]
Revised: [Equation]

Original: Ll =
Revised: ≈

Original: [Equation]
Revised: [Equation]

Original:  p ( z l | z l − 1 ) , where the mean and covariance of p ( z t | x 1: T ) and p ( z l | z l − 1 ) are obtained by marginalizing over the multivariate distributions p ( Z | X ) and p gen ( Z ) , respectively, yielding E [ z t | x 1: T ] , E
Revised: Ll =1 p ( z l | z l − 1 ) , where the mean and covariance of p ( z t | x 1: T ) and p ( z l | z l − 1 ) are obtained by marginalizing over the multivariate distributions p ( Z | X ) and p gen ( Z ) , respectively, yielding E [ z t | x 1: T ] , E

Original: [ z l | z l − 1 ] , and covariance matrices Var( z t | x 1: T ) and
Revised: [ z l | z l − 1 ] , and covariance matrices Var( z t | x 1: T ) and

Original: Var( z l | z l − 1 ) .
Revised: Var( z l | z l − 1 ) .

Original: Supplementary eq.
Revised: Supplementary eq.

Original: 52 may then be numerically approximated through Monte Carlo sampling (Hershey & Olsen, 2007) by
Revised: 52 may then be numerically approximated through Monte Carlo sampling (Hershey & Olsen, 2007) by

Original: [Equation]
Revised: [Equation]

Original: Alternatively, there is also a variational approximation of eq.
Revised: Alternatively, there is also a variational approximation of eq.

Original: 52 available (Hershey & Olsen,
Revised: 52 available (Hershey & Olsen,

Original: [Equation]
Revised: [Equation]

Original: where the KL divergences in the exponentials are among Gaussians for which we have an analytical expression.
Revised: where the KL divergences in the exponentials are among Gaussians for which we have an analytical expression.

Original: 6.1. More details on benchmark tasks and model comparisons
Revised: 6.1. More details on benchmark tasks and model comparisons

Original: We compared the performance of our rPLRNN to the other models summarized in Suppl.
Revised: We compared the performance of our rPLRNN to the other models summarized in Suppl.

Original: Tableon the following three benchmarks requiring long short-term maintenance of information (Talathi & Vartak (2016); Hochreiter & Schmidhuber (1997)): 1) The addition problem of time length T consists of 100 000 training and 10 000 test samples of 2 × T input series S =
Revised: Tableon the following three benchmarks requiring long short-term maintenance of information (Talathi & Vartak (2016); Hochreiter & Schmidhuber (1997)): 1) The addition problem of time length T consists of 100 000 training and 10 000 test samples of 2 × T input series S =

Original: { s 1 , . . .
Revised: { s 1 , . . .

Original: , s T } , where entries s 1 , : ∈ [0 , 1] are drawn from a uniform random distribution and s 2 , : ∈ { 0 , 1 } contains zeros except for two indicator bits placed randomly at times t 1 < 10 and t 2 < T/ 2 .
Revised: , s T } , where entries s 1 , : ∈ [0 , 1] are drawn from a uniform random distribution and s 2 , : ∈ { 0 , 1 } contains zeros except for two indicator bits placed randomly at times t 1 < 10 and t 2 < T/ 2 .

Original: Constraints on t 1 and t 2 are chosen such that every trial requires a long memory of at least T/ 2 time steps.
Revised: Constraints on t 1 and t 2 are chosen such that every trial requires a long memory of at least T/ 2 time steps.

Original: At the last time step T , the target output of the network is the sum of the two inputs in s 1 , : indicated by the 1-entries in s 2 , : , x target T = s 1 ,t 1 + s 1 ,t 2 . 2)
Revised: At the last time step T , the target output of the network is the sum of the two inputs in s 1 , : indicated by the 1-entries in s 2 , : , x target T = s 1 ,t 1 + s 1 ,t 2 . 2)

Original: The multiplication problem is the same as the addition problem, only that the product instead of the sum has to be produced by the RNN as an output at time T , x target T = s 1 ,t 1 · s 1 ,t 2 . 3)
Revised: The multiplication problem is the same as the addition problem, only that the product instead of the sum has to be produced by the RNN as an output at time T , x target T = s 1 ,t 1 · s 1 ,t 2 . 3)

Original: The MNIST dataset (LeCun et al., 2010) consists of 60 000 training and 10 000 28 × 28 test images of hand written digits.
Revised: The MNIST dataset (LeCun et al., 2010) consists of 60 000 training and 10 000 28 × 28 test images of hand written digits.

Original: To make this a time series problem, in sequential MNIST the images are presented sequentially, pixel-by-pixel, scanning lines from upper left to bottom-right, resulting in time series of ﬁxed length T = 784 .
Revised: To make this a time series problem, in sequential MNIST the images are presented sequentially, pixel-by-pixel, scanning lines from upper left to bottom-right, resulting in time series of ﬁxed length T = 784 .

Original: For training on the addition and multiplication problems, the mean squared-error loss across R samples, L = 1 R (cid:80) Rn =1 (cid:16) ˆ x ( n ) T − x ( n ) T (cid:17) 2 , between estimated and actual outputs was used, while the cross-entropy loss L = (cid:80) Rn =1 (cid:16) − (cid:80) 10 i =1 x ( n ) i,T log(ˆ p ( n ) i,T ) (cid:17) was employed for sequential MNIST, where
Revised: For training on the addition and multiplication problems, the mean squared-error loss across R samples, L = 1 R (cid:80) Rn =1 (cid:16) ˆ x ( n ) T − x ( n ) T (cid:17) 2 , between estimated and actual outputs was used, while the cross-entropy loss L = (cid:80) Rn =1 (cid:16) − (cid:80) 10 i =1 x ( n ) i,T log(ˆ p ( n ) i,T ) (cid:17) was employed for sequential MNIST, where

Original: [Equation]
Revised: [Equation]

Original: with x i,t ∈ { 0 , 1 } , (cid:80) i x i,t = 1 .
Revised: with x i,t ∈ { 0 , 1 } , (cid:80) i x i,t = 1 .

Original: We remark that as long as the observation model takes the form of a generalized linear model (Fahrmeir & Tutz, 2001), as assumed here, meaning may be assigned to the latent states z m by virtue of their association with speciﬁc sets of observations x n through the factor loading matrix B .
Revised: We remark that as long as the observation model takes the form of a generalized linear model (Fahrmeir & Tutz, 2001), as assumed here, meaning may be assigned to the latent states z m by virtue of their association with speciﬁc sets of observations x n through the factor loading matrix B .

Original: This adds another layer of model interpretability (besides its accessibility in DS terms).
Revised: This adds another layer of model interpretability (besides its accessibility in DS terms).

Original: [Table]
Revised: [Table]

Original: The large error bars in Fig.
Revised: The large error bars in Fig.

Original: 2 at the transition from good to bad performance result from the fact that the networks mostly learn these tasks in an all-or-none fashion.
Revised: 2 at the transition from good to bad performance result from the fact that the networks mostly learn these tasks in an all-or-none fashion.

Original: While the rPLRNN in general outperformed the pure initialization-based models (iRNN, npRNN, iPLRNN), conﬁrming that a manifold attractor subspace present at initialization may be lost throughout training, we conjecture that this difference in performance will become even more pronounced as noise levels or task complexity increase [Table]
Revised: While the rPLRNN in general outperformed the pure initialization-based models (iRNN, npRNN, iPLRNN), conﬁrming that a manifold attractor subspace present at initialization may be lost throughout training, we conjecture that this difference in performance will become even more pronounced as noise levels or task complexity increase.

Original: 6.1. More details on single neuron model
Revised: 6.1. More details on single neuron model

Original: The neuron model used in section 4.2 is described by
Revised: The neuron model used in section 4.2 is described by

Original: [Equation]
Revised: [Equation]

Original: where C m refers to the neuron’s membrane capacitance, the g • to different membrane conductances,
Revised: where C m refers to the neuron’s membrane capacitance, the g • to different membrane conductances,

Original: E • to the respective reversal potentials, and m , h , and n are gating variables with limiting values given by
Revised: E • to the respective reversal potentials, and m , h , and n are gating variables with limiting values given by

Original: [Equation]
Revised: [Equation]

Original: Different parameter settings in this model lead to different dynamical phenomena, including regular spiking, slow bursting or chaos (see Durstewitz (2009) for details).
Revised: Different parameter settings in this model lead to different dynamical phenomena, including regular spiking, slow bursting or chaos (see Durstewitz (2009) for details).

Original: Parameter settings used here were: C m = 6 µ F , g L = 8 mS , E L = − 80 mV , g Na = 20 mS , E Na = 60 mV , V hNa = − 20 mV , k Na = 15 , g K = 10 mS , E K = − 90 mV , V hK = − 25 mV , k K = 5 , τ n = 1 ms , g M = 25 mS ,
Revised: Parameter settings used here were: C m = 6 µ F , g L = 8 mS , E L = − 80 mV , g Na = 20 mS , E Na = 60 mV , V hNa = − 20 mV , k Na = 15 , g K = 10 mS , E K = − 90 mV , V hK = − 25 mV , k K = 5 , τ n = 1 ms , g M = 25 mS ,

Original: V hM = − 15 mV , k M = 5 , τ h = 200 ms , g NMDA = 10 .
Revised: V hM = − 15 mV , k M = 5 , τ h = 200 ms , g NMDA = 10 .

Original: 2 mS .
Revised: 2 mS .

Original: [Figure]
Revised: [Figure]

Original: Figure S is not a good measure for DS reconstruction.
Revised: [Figure] IGURES [Figure]

Original: A ) Time graph (top) and state space(bottom) for the single neuronmodel (see section 4.2 and Suppl. 6.1.8) with parameters in the chaotic regime (blue curves) and with simple ﬁxed point dynamics in the limit (red line).
Revised: graph top state space neuron with parameters in the chaotic regime (blue curves) and with simple ﬁxed point dynamics in the limit (red line).

Original: Although the system has vastly different limiting behaviors (attractor geometries) in these two cases, as visualized in the state space, the agreement in time series initially seems to indicate a perfect ﬁt.
Revised: Although the system has vastly different limiting behaviors (attractor geometries) in these two cases, as visualized in the state space, the agreement in time series initially seems to indicate a perfect ﬁt.

Original: B )
Revised: B )

Original: Same as in A ) for two trajectories drawn from exactly the same DS (i.e., same parameters) with slightly different initial conditions.
Revised: Same as in A ) for two trajectories drawn from exactly the same DS (i e., same parameters) with slightly different initial conditions.

Original: Despite identical dynamics, the trajectories immediately diverge, resulting in a high MSE.
Revised: Despite identical dynamics, the trajectories immediately diverge, resulting in a high MSE.

Original: Dash-dotted grey lines in top graphs indicate the point from which onward the state space trajectories were depicted. 
Revised: Dash-dotted grey lines in top graphs indicate the point from which onward the state space trajectories were depicted. dynamics

Original: In general, both the standard and the oPLRNN tended to produce many ﬁxed point solutions.
Revised: In general, both the standard and the oPLRNN tended to produce many ﬁxed point solutions.

Original: In those cases where this was not the case, the standard PLRNN tended to reproduce only the fast components of the dynamics as in the example in C (in agreement with the results in
Revised: In those cases where this was not the case, the standard PLRNN tended to reproduce only the fast components of the dynamics as in the example in C (in agreement with the results in

Original: Figs.
Revised: Figs.

Original: 3 C & 3 E ) while the oPLRNN tended to capture only the slow components as in the example in B (as expected from the fact that the orthogonality constraint tends to produce solutions similar to those obtained for the regularized only
Revised: 3 C & 3 E ), while the oPLRNN tended to capture only the slow components as in the example in B (as expected from the fact that the orthogonality constraint tends to produce solutions similar to those obtained for the regularized states only, cf Fig. 3 E )