Original: Discovering Dynamical Parameters by Interpreting Echo State Networks
Revised: Discovering Dynamical Parameters by Interpreting Echo State Networks

Original: Abstract
Revised: Abstract

Original: Reservoir computing architectures known as echo state networks (ESNs) havebeen shown to have exceptional predictive capabilities when trained on chaoticsystems.
Revised: Reservoir computing architectures known as echo state networks (ESNs) have been shown to have exceptional predictive capabilities when trained on chaotic systems.

Original: However, ESN models are often seen as black-box predictors that lackinterpretability.
Revised: However, ESN models are often seen as black-box predictors that lack interpretability.

Original: We show that the parameters governing the dynamics of a complexnonlinear system can be encoded in the learned readout layer of an ESN.
Revised: We show that the parameters governing the dynamics of a complex nonlinear system can be encoded in the learned readout layer of an ESN.

Original: We canextract these dynamical parameters by examining the geometry of the readout layerweights through principal component analysis.
Revised: We can extract these dynamical parameters by examining the geometry of the readout layer weights through principal component analysis.

Original: We demonstrate this approach byextracting the values of three dynamical parameters ( σ , ρ , β ) from a dataset of
Revised: We demonstrate this approach by extracting the values of three dynamical parameters ( σ , ρ , β ) from a dataset of

Original: Lorenz systems where all three parameters are varying among different trajectories.
Revised: Lorenz systems where all three parameters are varying among different trajectories.

Original: Our proposed method not only demonstrates the interpretability of the ESN readoutlayer but also provides a computationally inexpensive, unsupervised data-drivenapproach for identifying uncontrolled variables affecting real-world data fromnonlinear dynamical systems.
Revised: Our proposed method not only demonstrates the interpretability of the ESN readout layer but also provides a computationally inexpensive, unsupervised data-driven approach for identifying uncontrolled variables affecting real-world data from nonlinear dynamical systems.

Original:  Introduction
Revised:  Introduction

Original: The analysis and modeling of complex nonlinear dynamics is a key problem across a broad range ofdisciplines in science and engineering [1].
Revised: The analysis and modeling of complex nonlinear dynamics is a key problem across a broad range of disciplines in science and engineering [1].

Original: While supervised deep learning approaches can provideexcellent predictive performance on dynamical systems [2–4], difﬁculties with interpretability limittheir usefulness in some scientiﬁc applications.
Revised: While supervised deep learning approaches can provide excellent predictive performance on dynamical systems [2–4], difﬁculties with interpretability limit their usefulness in some scientiﬁc applications.

Original: For example, consider the problem of identifyinguncontrolled dynamical parameters causing variations in the dynamics of a set of observed trajectories.
Revised: For example, consider the problem of identifying uncontrolled dynamical parameters causing variations in the dynamics of a set of observed trajectories.

Original: Directly learning a black-box predictor for each trajectory does not provide any insight into howthe dynamics are changing between trajectories.
Revised: Directly learning a black-box predictor for each trajectory does not provide any insight into how the dynamics are changing between trajectories.

Original: Potential approaches for solving this probleminclude methods that attempt to directly recover the symbolic governing equations [5–9], which oftenrequire a large library of possible terms along with assumptions about the sparsity of the equations,and Koopman operator-based methods [10–13] that map the nonlinear dynamics to an interpretablelinear system but struggle with chaotic dynamics.
Revised: Potential approaches for solving this problem include methods that attempt to directly recover the symbolic governing equations [5–9] which often require a large library of possible terms along with assumptions about the sparsity of the equations, and Koopman operator-based methods [10–13] that map the nonlinear dynamics to an interpretable linear system but struggle with chaotic dynamics.

Original: Unsupervised learning approaches that use deepautoencoder architectures to provide interpretability have also been investigated [14, 15].
Revised: Unsupervised learning approaches that use deep autoencoder architectures to provide interpretability have also been investigated [14, 15].

Original: We propose an alternative approach for discovering varying dynamical parameters that is based onecho state networks (ESNs), a type of reservoir computing architecture [16, 17].
Revised: We propose an alternative approach for discovering varying dynamical parameters that is based on echo state networks (ESNs), a type of reservoir computing architecture [16, 17].

Original: ESNs provide afast and easy way to build predictive models for dynamical systems [18–20] and have been shownto perform remarkably well when trained to forecast complex nonlinear dynamics exhibiting highdimensional chaos [4, 19, 21].
Revised: ESNs provide a fast and easy way to build predictive models for dynamical systems [18–20] and have been shown to perform remarkably well when trained to forecast complex nonlinear dynamics exhibiting highdimensional chaos [4, 19, 21].

Original: While superﬁcially similar to recurrent architectures in deep learning,
Revised: While superﬁcially similar to recurrent architectures in deep learning,

Original: ESNs are inexpensive to train and do not suffer from exploding or vanishing gradients due to theiruse of a large sparse reservoir with ﬁxed weights [4, 18].
Revised: ESNs are inexpensive to train and do not suffer from exploding or vanishing gradients due to their use of a large sparse reservoir with ﬁxed weights [4, 18].

Original: Correctly constructed ESNs also exhibitthe echo state property [16, 22], which provides many nice theoretical guarantees and allows for the
Revised: Correctly constructed ESNs also exhibit the echo state property [16, 22], which provides many nice theoretical guarantees and allows for the

Original: ESN to be effectively trained by only ﬁtting the output weights of the ﬁnal readout layer via linearregression.
Revised: ESN to be effectively trained by only ﬁtting the output weights of the ﬁnal readout layer via linear regression.

Original: However, ESNs are generally considered black-box predictors that have the same issues with interpretability as deep learning approaches.
Revised: However, ESNs are generally considered black-box predictors that have the same issues with interpretability as deep learning approaches.

Original: Our work demonstrates that this is not the case.
Revised: Our work demonstrates that this is not the case.

Original: By sharingthe ﬁxed weights across a series of separately trained ESNs, we can interpret the learned outputweights from the readout layer as a representation of the system dynamics and use manifold learningto extracted the varying dynamical parameters affecting the data.
Revised: By sharing the ﬁxed weights across a series of separately trained ESNs, we can interpret the learned output weights from the readout layer as a representation of the system dynamics and use manifold learning to extracted the varying dynamical parameters affecting the data.

Original: Using the chaotic Lorenz system[23] as an example, we show that this approach is able to quickly and accurately learn high-qualitypredictive models and then extract an embedding which encodes three distinct varying parametersthat affect the dynamics of the system.
Revised: Using the chaotic Lorenz system [23] as an example, we show that this approach is able to quickly and accurately learn high-quality predictive models and then extract an embedding which encodes three distinct varying parameters that affect the dynamics of the system.

Original:  Methods
Revised:  Methods

Original: To produce a set of interpretable representations of the system dynamics, we train a series of echostate networks (ESNs) that share the same reservoir and input weights.
Revised: To produce a set of interpretable representations of the system dynamics, we train a series of echo state networks (ESNs) that share the same reservoir and input weights.

Original: Each ESN is trained on adistinct trajectory in the dataset and, after training, produces a linear readout layer which we useas our representation of the dynamics.
Revised: Each ESN is trained on a distinct trajectory in the dataset and, after training, produces a linear readout layer which we use as our representation of the dynamics.

Original: We then analyze the geometry of the readout layer weightsusing a manifold learning method—in our case, principal component analysis (PCA)—and extractparameters that account for the varying dynamics among the trajectories in the dataset.
Revised: We then analyze the geometry of the readout layer weights using a manifold learning method—in our case, principal component analysis (PCA)—and extract parameters that account for the varying dynamics among the trajectories in the dataset.

Original: 2.1 ESN training and prediction
Revised: 2.1 ESN training and prediction

Original: ESNs are reservoir computing architectures that consist of a large, ﬁxed sparse reservoir A ∈ R n × nwith hidden size n , a ﬁxed set of input weights W in ∈ R n × d , and a trained set of output or readoutweights W out ∈ R d × n .
Revised: ESNs are reservoir computing architectures that consist of a large, ﬁxed sparse reservoir A ∈ R n × n with hidden size n , a ﬁxed set of input weights W in ∈ R n × d , and a trained set of output or readout weights W out ∈ R d × n .

Original: A standard discrete time ESN can model a d -dimensional dynamical systemd u ( t ) d t = F
Revised: A standard discrete time ESN can model a d -dimensional dynamical system

Original: [Equation]
Revised: [Equation]

Original: ( u ( t )) (1) with dynamical parameters θ and states u ( t ) 
Revised: with dynamical parameters θ and states u ( t ) ∈

Original: ∈ R
Revised: R

Original: d discretized as a time series { u t 0 , u t 0 +∆ t , . . .
Revised: d discretized as a time series { u t 0 , u t 0 +∆ t , . . .

Original: }with a ﬁxed time step ∆ t .
Revised: } with a ﬁxed time step ∆ t .

Original: Training —To train the output weights W out , we ﬁrst feed a training trajectory { u t } T train t = t 0 into the
Revised: Training —To train the output weights W out , we ﬁrst feed a training trajectory { u t } T train t [Equation] t 0 into the

Original: ESN in a recurrent fashion
Revised: ESN in a recurrent fashion

Original: [Equation]
Revised: [Equation]

Original: producing a series of hidden reservoir states { r t 0 , r t 0 +∆ t , . . .
Revised: producing a series of hidden reservoir states { r t 0 , r t 0 +∆ t , . . .

Original: } where each r t ∈ R n and r t 0 = 0is initialized as the zero vector.
Revised: } where each r t ∈ R n and r t 0 =is initialized as the zero vector.

Original: The output weights W out are then ﬁtted using a (regularized)linear regression to map the hidden states r t back to the physical states u t such that u t ≈ W out r t ,corresponding to the ﬁnal readout layer of the ESN.
Revised: The output weights W out are then ﬁtted using a (regularized) linear regression to map the hidden states r t back to the physical states u t such that u t ≈ W out r t , corresponding to the ﬁnal readout layer of the ESN.

Original: Because the training only involves passing theinputs through the ESN once and then ﬁtting the ﬁnal readout layer using linear regression, thisprocess is both fast and computationally inexpensive.
Revised: Because the training only involves passing the inputs through the ESN once and then ﬁtting the ﬁnal readout layer using linear regression, this process is both fast and computationally inexpensive.

Original: Prediction —To predict the future states of a test trajectory { u t } T test t = t 0 , the new trajectory is fedrecurrently into the ESN as during training (2), and then future physical states ˆ u t =
Revised: Prediction —To predict the future states of a test trajectory { u t } T test t = t 0 , the new trajectory is fed recurrently into the ESN as during training (2), and then future physical states ˆ u t =

Original: W out r t fort > T test are computed using the output weights from the readout layer.
Revised: W out r t for t > T test are computed using the output weights from the readout layer.

Original: The predicted states ˆ u t arethen fed back into the recurrent architecture to obtain subsequent hidden states
Revised: The predicted states ˆ u t are then fed back into the recurrent architecture to obtain subsequent hidden states

Original: [Equation]
Revised: [Equation]

Original: 2.2 Training ESNs with unique and comparable readout layers
Revised: 2.2 Training ESNs with unique and comparable readout layers

Original: Because our approach treats the trained output weights W out from the readout layer as a representation of the system dynamics, we require W out to be comparable across ESNs trained on distincttrajectories with varying dynamics.
Revised: Because our approach treats the trained output weights W out from the readout layer as a representation of the system dynamics, we require W out to be comparable across ESNs trained on distinct trajectories with varying dynamics.

Original: To ensure this, we use the same ﬁxed reservoir and input weightsfor all our ESNs and impose L 2 regularization on the linear regression (i.e. ridge regression) whenﬁtting W out .
Revised: To ensure this, we use the same ﬁxed reservoir and input weights for all our ESNs and impose L 2 regularization on the linear regression (i e. ridge regression) when ﬁtting W out .

Original: Along with proper initialization of the ESN to ensure the echo state property (ESP)is fulﬁlled [16, 22], these conditions also maintain uniqueness among the learned readout layerscorresponding to different dynamics.
Revised: Along with proper initialization of the ESN to ensure the echo state property (ESP) is fulﬁlled [16, 22], these conditions also maintain uniqueness among the learned readout layers corresponding to different dynamics.

Original: The ESP guarantees that each input sequence { u t } T − 1 t = −∞corresponds to a unique reservoir state r T .
Revised: The ESP guarantees that each input sequence { u t } T − 1 t = −∞ corresponds to a unique reservoir state r T .

Original: Then, because the dynamical system is Markovian (1),each set of dynamical parameters θ and physical state u T have a unique input sequence history and socorrespond to a unique r T .
Revised: Then, because the dynamical system is Markovian (1), each set of dynamical parameters θ and physical state u T have a unique input sequence history and so correspond to a unique r T .

Original: Therefore, the mapping r T → u T , which is approximated by the readoutlayer, is also unique for each θ .
Revised: Therefore, the mapping r T → u T , which is approximated by the readout layer, is also unique for each θ .

Original: Finally, by using ridge regression to ﬁt the readout layer—a commonpractice for training ESNs—we obtain a unique set of output weights W out for each θ .
Revised: Finally, by using ridge regression to ﬁt the readout layer—a common practice for training ESNs—we obtain a unique set of output weights W out for each θ .

Original: 2.3 Extracting dynamical parameters using manifold learning
Revised: 2.3 Extracting dynamical parameters using manifold learning

Original: Once we have the dynamics encoded in the output weights W out  we can analyze the geometry ofthis representation by performing manifold learning.
Revised: Once we have the dynamics encoded in the output weights W out , we can analyze the geometry of this representation by performing manifold learning.

Original: In our experiments we found that applying
Revised: In our experiments, we found that applying

Original: PCA to the ﬂattened output weights was enough to extract a reasonable low-dimensional embeddingthat encodes the dynamical parameters θ .
Revised: PCA to the ﬂattened output weights was enough to extract a reasonable low-dimensional embedding that encodes the dynamical parameters θ .

Original:  Lorenz System Experiment
Revised:  Lorenz System Experiment

Original: The Lorenz system [23] is a three-dimensional chaotic dynamical system governed by
Revised: The Lorenz system [23] is a three-dimensional chaotic dynamical system governed by

Original: [Equation]
Revised: [Equation]

Original: (4) with the parameters σ , ρ , β traditionally chosen to be 10 , 28 , 8 / 3 , respectively.
Revised: with the parameters σ , ρ , β traditionally chosen to be 10 , 28 , 8 / 3 , respectively.

Original: For our dataset, wegenerate 1000 trajectories each with 300,000 time steps ( ∆ t = 0 . 02 ) for training and a subsequent50,000 time steps for testing.
Revised: For our dataset, we generate 1000 trajectories each with 300,000 time steps ( ∆ t = 0 . 02 ) for training and a subsequent 50,000 time steps for testing.

Original: For each trajectory, we independently sample the three dynamical parameters σ , ρ , β from uniform distributions that cover ± 5% of their standard values to mimicuncontrolled variations in the dynamics.
Revised: For each trajectory, we independently sample the three dynamical parameters σ , ρ , β from uniform distributions that cover ± 5% of their standard values to mimic uncontrolled variations in the dynamics.

Original: Using ESN initialization hyperparameters suggested by
Revised: Using a sparse reservoir (with size 300, degree 6, and a spectral radius of 0 .

Original: Haluszczynski and Räth [20] and ridge regression regularization parameter γ = 0 .
Revised: 1 as suggested by [20]), an input weight scaling of 0 . 1 , as well as a ridge regression regularization parameter γ = 0 .

Original: 1 , we achievelong-term stability and good prediction performance (Fig. 1) with a mean prediction horizon of 3 . 2 τ .
Revised: 1 , we achieve long-term stability and good prediction performance (Fig. 1) with a mean prediction horizon of 3 . 2 τ .

Original: Here, τ = Λ − 1max refers to the Lyapunov time, i.e. the inverse of the maximum Lyapunov exponentΛ max ,of the chaotic system which characterizes how quickly two similar trajectories diverge.
Revised: Here, τ = Λ − 1max refers to the Lyapunov time of the chaotic system, i.e. the inverse of the maximum Lyapunov exponent Λ max , which characterizes how quickly two similar trajectories diverge.

Original: Applying PCA to the trained output weights W out , we are able to extract a three-dimensionalembedding that directly corresponds to the varying dynamical parameters σ , ρ , β of the Lorenzsystem (Fig. 2(a)–(c)).
Revised: Applying PCA to the trained output weights W out , we can identify the 3 relevant components based on their explained variance ratios ( 0 . 750 , 0 . 165 , 0 . 062 ) using a total explained variance threshold of 0 . These PCA components provide a three-dimensional embedding that directly corresponds to the varying dynamical parameters σ , ρ , β of the Lorenz system (Fig. 2(a)–(c)).

Original: We also obtain high-quality linear ﬁts for the parameters σ , ρ , β from theextracted three-dimensional embedding (Fig. 2(d)–(f)) with R 2 correlation coefﬁcients of 0 .
Revised: We also obtain high-quality linear ﬁts for the parameters σ , ρ , β from the extracted three-dimensional embedding (Fig. 2(d)–(f)) with R 2 correlation coefﬁcients of 0 .

Original: 996 , 0 . 925 , respectively.
Revised: 996 , 0 . 925 , respectively.

Original:  Conclusion
Revised:  Conclusion

Original: [Figure] We have empirically conﬁrmed that parameters governing the dynamics of a nonlinear system areencoded in the readout layer of a trained ESN and have demonstrated a method for extracting anembedding of the varying dynamical parameters using PCA.
Revised: We have empirically conﬁrmed that parameters governing the dynamics of a nonlinear system are encoded in the readout layer of a trained ESN and have demonstrated a method for extracting an embedding of the varying dynamical parameters using PCA.

Original: Our proposed method for analyzing datafrom dynamical systems, including chaotic systems, provides a fast and computationally inexpensiveapproach for both predicting future states and understanding the uncontrolled variables causingvariations in the dynamics.
Revised: Our proposed method for analyzing data from dynamical systems, including chaotic systems, provides a fast and computationally inexpensive approach for both predicting future states and understanding the uncontrolled variables causing variations in the dynamics.

Revised: While our method only extracts a latent embedding of the dynamical parameters without an explicit symbolic interpretation, the geometry of this embedding is already very informative on its own: trajectories with similar dynamics are embedded nearby; each relevant component of the embedding corresponds to a varying parameter; and the intrinsic dimensionality of the embedding manifold (i.e. the number of relevant components) gives the number of varying parameters.

Original: Given the extracted embedding of the dynamical parameters, it shouldalso be feasible to approximately reconstruct ESN output weights as a function of the identiﬁeddynamical parameters, effectively creating a tunable model for the dynamics learned from the data.
Revised: Given the extracted embedding, it should also be feasible to approximately reconstruct ESN output weights as a function of the identiﬁed dynamical parameters, effectively creating a tunable model for the dynamics learned from the data.

Original: This would aid in directly interpreting unknown dynamical parameters extracted using our method,so we hope to  explore this direction in the near future.
Revised: This would aid in directly interpreting unknown dynamical parameters extracted using our method, so we hope to further explore this direction in the near future.

Original: Also, because our approach builds on topof standard ESN training methods, it would be straightforward to adapt for any applications where ESNs are currently being used purely for prediction.
Revised: Also, because our approach builds on top of standard ESN training methods, it would be straightforward to adapt for any applications where ESNs are currently being used purely for prediction.

Original: These applications include modeling EEGsignals [24], making atmospheric climate predictions [25], and scaling up ESNs to model arbitrarilyhigh-dimensional chaotic attractors [21].
Revised: These applications include modeling EEG signals [24], making atmospheric climate predictions [25], and scaling up ESNs to model high-dimensional chaotic attractors [21].

Original: As such, we plan to test our method on a wider variety ofexamples, including systems exhibiting spatiotemporal chaos.
Revised: As such, we plan to test our method on a wider variety of examples, including systems exhibiting spatiotemporal chaos, as well as investigate the effects of noise on our approach.

Revised: Acknowledgments and Disclosure of Funding

Revised: We would like to acknowledge useful discussions with Rumen Dangovski, Samuel Kim, Andrew Ma, and Charlotte Loh, and Dr. John Rickert.

Revised: This research is supported in part by the U.S. Department of Defense through the National Defense Science & Engineering Graduate Fellowship (NDSEG)

Revised: Program and the Center for Excellence in Education (CEE) through the Research Science Institute (RSI) summer program.

Revised: This work is further supported in part by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artiﬁcial Intelligence and

Revised: Fundamental Interactions, http://iaifi.org/ ).

Revised: It is also based upon work supported in part by the U.S. Army Research Ofﬁce through the Institute for Soldier Nanotechnologies at MIT, under

Revised: Collaborative Agreement Number W911NF-18-2-