A K OOPMAN A PPROACH TO U NDERSTANDING S EQUENCE N EURAL M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning models are often treated as “black boxes”.
Existing approaches for understanding the decision mechanisms of neural networks provide limited explanations or depend on local theories, such as ﬁxed-point analysis.
Recently, a data-driven framework based on Koopman theory was developed for the analysis of nonlinear dynamical systems.
In this paper, we introduce a new approach to understanding trained sequence neural models: the Koopman Analysis of Neural Networks ( KANN ) method.
At the core of our method lies the Koopman operator, which is a linear operator that encodes the dominant features of the network latent dynamics.
In practice, we compute this operator by representing the hidden states of the network in a basis, and the operator is deﬁned to be the linear ﬁt in that new space.
Since it is a linear operator, we can study its eigenvectors and eigenvalues, and we observe they facilitate understanding: in the sentiment analysis problem, the eigenvectors highlight positive and negative n -grams; and, in the ECG classiﬁcation challenge, the eigenvectors capture the salient features of normal beat signals.
I NTRODUCTION
Understanding the inner workings of predictive models is an essential requirement in many ﬁelds across science and engineering.
This need is even more important nowadays with the emergence of neural networks whose visualization and interpretation is inherently challenging.
Indeed, modern computational neural models often lack a commonly accepted knowledge regarding their governing mathematical principles.
Consequently, while deep neural networks may achieve remarkable results on various complex tasks, explaining their underlying decision mechanisms remains a challenge.
The goal of this paper is to help bridge this gap by proposing a new framework for the approximation, reasoning, and understanding of sequence neural models.
Sequence models are designed to handle time series data originating from images, text, audio, and other sources of information.
One approach to analyzing sequence neural networks is through the theory and practice of dynamical systems (Doya, 1993a; Pascanu et al., 2013).
For instance, the temporal asymptotic behavior of a dynamical system can be described using the local analysis of its attractor states (Strogatz, 2018).
Similarly, recurrent models have been investigated in the neighborhood of their ﬁxed points (Sussillo & Barak, 2013), leading to work that interprets trained
RNNs for tasks such as sentiment analysis (Maheswaranathan et al., 2019).
However, the local nature of these methods is a limiting factor which may lead to inconsistent results.
Speciﬁcally, their approach is based on ﬁxed-point analysis which allows to study the dynamical system in the neighborhood of a ﬁxed-point.
In contrast, our approach is global—it does not depend on a set of ﬁxed-points, and it facilitates the exploration of the dynamics near and further away from ﬁxed points.
Over the past few years, a family of data-driven methods was developed, allowing to analyze complex dynamical systems based on Koopman (1931) theory.
These methods exploit a novel observation by which nonlinear systems may be globally encoded using inﬁnite-dimensional but linear Koopman operators.
In practice, Koopman-based approaches are lossy as they compute a ﬁnite-dimensional approximation of the full operator.
Nevertheless, it has been shown in the ﬂuid dynamics (Azencot et al., 2020; Mezi´c, 2005) and geometry processing (Sharma & Ovsjanikov, 2020; Ovsjanikov et al., 2012) communities that the dominant features of general nonlinear dynamical systems can be captured via a single matrix per system, allowing e.g., to align time series data (Rahamim & Talmon, 2021).
Thus, we pose the following research question: can we design and employ a Koopman-based approach to analyze and develop a fundamental understanding of deep neural models?
Given a trained sequence neural network and a procedure to extract its hidden states, our Koopmanbased method generates a moderate size matrix which faithfully describes the dynamics in the latent space.
Unlike existing work, our approach is global and independent of a particular latent sample, and thus it can be virtually applied to any hidden state.
A key advantage of our framework is that we can directly employ linear analysis tools on the approximate Koopman operator to reason about the associated neural network.
In particular, we show that the eigenvectors and eigenvalues of the
Koopman matrix are instrumental for understanding the decision mechanisms of the model.
For instance, we show in our results that the dominant eigenvectors carry crucial semantic knowledge related to the problem at hand.
Moreover, the eigenvalues represent the memory of the network as they provide a timestamp for the temporal span of the respective eigenvectors.
Finally and most importantly, Koopman-based methods such as ours are backed by rich theory and practice, allowing us to exploit the recent advances in Koopman inspired techniques for the purpose of developing a comprehensive understanding of sequence neural networks.
Thus, the key contribution in this work is the novel application of Koopman-based methods for understanding sequential models, and the extraction of high-level interpretable and insightful understandings on the trained networks.
We focus our investigation on two learning tasks: sentiment analysis and electrocardiogram (ECG) classiﬁcation.
We will identify four eigenvectors in the sentiment analysis model whose roles are to highlight: positive words, negative words, positive pairs (e.g., “not bad”), and negative pairs.
In addition, we demonstrate that the eigenvectors in the ECG classiﬁcation task naturally identify dominant features in normal beat signals and encode them.
Speciﬁcally, we show that four Koopman eigenvectors accurately capture the local extrema points of normal beat signals.
These extrema points are fundamental in deciding whether a signal is normal or anomalous.
Our results reinforce that the network indeed learns a robust representation of normal beat signals.
Then, we will verify that the main components of the nonlinear network dynamics can be described using our Koopman matrices by measuring the difference in accuracy results, and the relative error in predicted states.
Further, we provide additional results and comparisons in the supplementary material.
Given the versatility of our framework and its ease of use, we advocate its utility in the analysis and understanding of neural networks, and we believe it may also affect the design and training of deep models in the future.
R ELATED W ORK
Recurrent Neural Networks (RNN) and Dynamical Systems (DS).
Fully connected recurrent neural networks are universal approximators of arbitrary dynamical systems (Doya, 1993b).
Unfortunately, RNNs are well-known to be difﬁcult to train (Bengio et al., 1993; Pascanu et al., 2013), and several methods adopt a DS perspective to improve training via gradient clipping (Pascanu et al., 2013), and constraining the weights (Erichson et al., 2021), among other approaches.
Overall, it is clear that dynamical systems are fundamental in investigating and developing recurrent networks.
Understanding RNN.
Establishing a deeper understanding of recurrent networks is a long standing challenge in machine learning.
To this end, Karpathy et al.
(2015) follow the outputs of the model to identify units which track brackets, line lengths, and quotes.
Recently, Chefer et al.
(2020) proposed an approach for computing relevance scores of transformer networks.
Perhaps mostly related to our approach is the analysis of recurrent models around their ﬁxed points (Sussillo & Barak, 2013).
This approach revealed low-dimensional attractors in the sentiment analysis task (Maheswaranathan et al., 2019), which allowed to deduce simple explanations of the decision mechanisms of the associated models.
Our work generalizes the approach of Sussillo & Barak (2013) in that it yields global results about the dynamics, and it introduces several novel features.
We provide a more detailed comparison between our method and theirs in Sec. 4.
Koopman-based Neural Networks.
Recently, several techniques that combine neural networks and Koopman theory were proposed, mostly in the context of predicting nonlinear dynamics.
For example, Takeishi et al.
; Morton et al.
(2018) optimize the residual sum of squares of the predictions the operator makes, Lusch et al.
; Erichson et al.
; Azencot et al.
(2020) design dynamic autoencoders whose central component is linear and may be structured, Li et al.
(2020) employ graph networks, and Mardt et al.
(2018) use a variational approach on Markov processes.
A recent line of work aims at exploiting tools from Koopman theory to analyze and improve the training process of neural networks (Dietrich et al., 2020; Dogra & Redman, 2020;
Manojlovi´c et al., 2020).
To the best of our knowledge, our work is ﬁrst to employ a Koopman-based method towards the analysis and understanding of trained neural networks.
M ETHOD
In what follows, we present our method for analyzing and understanding sequence neural models.
Importantly, while we mostly discuss and experiment with recurrent neural networks, our approach is quite general and applicable to any model whose inner representation is a time series.
We consider neural models that take input instances x t ∈ R m at time t and compute
[Equation]
where h t ∈ R k is a (hidden) state that represents the latent dynamics, and F is some nonlinear function that pushes states through time.
In our analysis, we use only the hidden states set and discard the time series input.
Thus, our method is a “white-box” approach as we assume access to { h t } , which is typically possible in most day-to-day scenarios.
Importantly, all recurrent models including vanilla RNN (Elman, 1990), LSTM (Hochreiter & Schmidhuber, 1997), and GRU (Cho et al., 2014), as well as Attention Models (Bahdanau et al., 2015; Vaswani et al., 2017), and Residual neural networks (He et al., 2016) exhibit the structure of Eq.
3. E SSENTIALS OF K OOPMAN THEORY
Our approach is based on Koopman (1931) theory which was developed for dynamical systems.
The key observation of Koopman was that a ﬁnite-dimensional nonlinear dynamics can be fully represented using an inﬁnite-dimensional but linear operator.
While the theoretical background is essential for developing a deep understanding of Koopman-based approaches, the practical aspects are more important to this work.
Thus, we brieﬂy recall the deﬁnition of the Koopman operator , and we refer the reader to other, comprehensive works on the subject (Singh & Manhas, 1993;
Eisner et al., 2015).
Formally, we assume a discrete-time dynamical system ϕ
acting on a compact, inner-product space M ⊂ R m ,
[Equation]
where t is an integer index representing discrete time.
The dynamics ϕ induces a linear operator K ϕ which we call the Koopman operator, and it is given by
[Equation]
where f : M → R is a scalar function in a bounded inner product space F .
It is easy to show that
K ϕ is linear due to the linearity of composition, i.e., given α, β ∈ R and f, g ∈ F , we obtain that
K ϕ ( αf + βg ) =
( αf + βg ) ◦ ϕ
= αf ◦ ϕ
+ βg ◦ ϕ
=
α K ϕ ( f ) + β K ϕ ( g ) .
We emphasize that while ϕ describes the system evolution, K ϕ is a transformation on the space of functions .
From a practical viewpoint, these functions may be interpreted as observations of the system, such as velocity, sea level, temperature, or hidden states in our setup.
To justify our use of Koopman theory and practice in the context of neural networks, we propose the following.
We interpret the input sequence { x t } as governed by some complex and unknown dynamics ϕ , i.e., x t +1 = ϕ
( x t ) for every t .
We emphasize that ϕ is different from F in Eq.
(1) by its deﬁnition of domain and range.
Then, the hidden states h t are ﬁnite samplings of observations of the system, namely, h t ≈ f t where f t : M → R is the true observation.
For instance, f t may be the smooth function cos( tz ) , whereas h t ∈ R k is its sampling at a ﬁnite set of points { z 1 , . . .
, z k } .
It follows that { h t } is subject to an approximate Koopman representation.
However, a fundamental challenge in facilitating Koopman theory in practice is the inﬁnite-dimensionality of K ϕ .
Recently, several data-driven methods were developed to produce a better approximate K ϕ using a moderatesize matrix C (Schmid, 2010; Ovsjanikov et al., 2012).
In particular, Koopman-based approaches have been proven instrumental in the analysis of ﬂuid dynamics data (Brunton et al., 2021), and for computing complex nonrigid isometric maps between shapes (Ovsjanikov et al., 2016).
Motivated by these empirical examples and their success, we will compute in this work approximate Koopman operator matrices C such that they encode the evolution of latent states { h t } .
3. A K OOPMAN - BASED METHOD
We denote by H ∈ R s × n × k a tensor of hidden state sequences, where s is the batch size, n is the sequence length and k is the hidden dimension.
The method we employ for computing the matrix C follows two simple steps: 1.
Represent the states using a basis B , and denote the resulting collection of spectral coefﬁcients by ˜ H .
Find the best linear transformation C which maps ˜ H t to ˜ H t +1 in the spectral domain, where ˜ H τ ∈ R s × k denotes the tensor of coefﬁcients from ˜ H at time τ .
To give a speciﬁc example of the general procedure we just described, we can choose the principal components b j , j = 1 , 2 , ... of the truncated SVD of the states H to be the basis in the ﬁrst step.
Then, the resulting basis elements are orthonormal, i.e., B T B = Id , where B = ( b j ) is the matrix of basis elements organized in its columns, and Id is the identity matrix.
The matrix C is obtained by solving the following least squares minimization
[Equation]
where · is matrix multiplication.
We note that the above scheme is a variant of the dynamic mode decomposition (Schmid, 2010), and the functional maps (Ovsjanikov et al., 2012) algorithms.
3. K OOPMAN - BASED PREDICTION
The inﬁnite-dimensional Koopman operator in Eq.
(3) describes the evolution of observable functions subject to the dynamics ϕ .
Similarly, our C matrices allow us to predict a future hidden state h t + from a given current state h t simply by multiplying C with the spectral coefﬁcients ˜ h t .
Namely,
[Equation]
We will mostly use Eq.
(6) to evaluate the validity of C in encoding the underlying dynamics based on the differences | H KANN t − H t | 2 F / | H t | 2 F for every admissible t , see Sec. 4.
3. K OOPMAN - BASED ANALYSIS
The key advantage of Koopman theory and practice is that linear analysis tools can be directly applied to study the behavior of the underlying dynamical system.
The tools we describe next form the backbone of our analysis framework, and our results are heavily based on these tools.
Separable dynamics.
If C ∈ R k × k admits an eigendecomposition, then the dynamics can be represented in a fully separable manner, where the eigenvectors of C propagate along the dynamics independently of the other eigenvectors, scaled by their respective eigenvalues.
Formally, we consider the eigenvalues λ j ∈ C and eigenvectors v j ∈ C k of C , i.e., it holds
that C v j = λ j v j
We assume that C is full-rank and thus V = ( v j ) forms a basis of R k , and similarly, U =
V − 1 is also a spanning basis.
In our setting, we call the rows of U the Koopman eigenvectors , and we represent any hidden state h t in this basis, similarly to Eq.
The projection of H onto U reads
[Equation]
Then, re-writing Eq.
(6) using the eigendecomposition of C = V · Λ · U yields the temporal trajectory of H t via ˆ H t +
=
H t +1 · B · V ≈ H t ·
B · C · V = H t · B · V · Λ = ˆ H t · Λ , where Λ is the diagonal matrix of eigenvalues, and the approximation is due to Eq.
The latter derivation yields
[Equation]
i.e., the linear dynamics matrix represented in the basis U is simply the diagonal matrix Λ , and thus
U may be viewed as a “natural” basis for encoding the dynamics.
Further, it directly follows that ˆ
H t + l
≈
ˆ
H t · Λ l , that is, the number of steps forward is determined by the eigenvalues power.
Memory horizon.
In addition to their role in describing the evolution of the hidden states, the eigenvalues naturally encode a memory horizon “timestamp” associated with each of the eigenvectors.
Speciﬁcally, if | λ j | < 1 , then its powers decay exponentially to zero.
We deﬁne the memory horizon of u j to be the decay time τ j ( (cid:15) ) of its associated eigenvalue, i.e.,
[Equation]
where 0 < (cid:15) (cid:28) 1 is a small threshold parameter.
We note that both log( (cid:15) ) and log( | λ j | ) are negative, and thus τ j ( (cid:15) ) > 0 .
Given a choice of (cid:15) , the eigenvector u j becomes almost insigniﬁcant to the dynamics in Eq.
(9) after τ j ( (cid:15) ) steps as it is scaled by (cid:15) =
| λ j | τ j ( (cid:15) ) .
In this context, eigenvectors whose eigenvalues satisfy | λ j | = 1 have inﬁnite memory, as powers of their eigenvalues simply rotate over the unit circle.
Finally, if | λ j | > 1 , its respective eigenvector leads to an unstable behavior since lim l →∞ | λ lj | = ∞ .
R ESULTS
In this study, we focus our exploration on the sentiment analysis and the ECG classiﬁcation problems.
Unless noted otherwise, we always compute C using the method in Sec 3.2, where the basis is given by the truncated SVD modes of the input hidden states, and C is the least squares estimation obtained from solving (4).
We ﬁrst provide our qualitative analysis in 4.1, and 4.2.
Then, we include in 4.3 a quantitative evaluation of KANN and its ability to encode the dynamics.
In Apps.
A, D, and E, we provide additional results, and we show that our method is robust to the choice of basis and network architecture.
Finally, we further use KANN to analyze the copy problem in App.
F, where our results outperform the baseline approach (Maheswaranathan et al., 2019).
4. S ENTIMENT ANALYSIS
We begin our qualitative study by considering the sentiment analysis task which was extensively explored in (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020).
Determining the sentiment of a document is an important problem which may be viewed as a binary classiﬁcation task.
We will use the IMDB reviews dataset, and we will embed the corpus of words to obtain a vector representation of text.
Given a review, the role of the network is to output whether it reﬂects a positive or negative opinion.
Adopting the setup of Maheswaranathan et al.
(2019), we use a word embedding of size 128 , and a GRU recurrent layer with a hidden size of 256 .
We train the model for 5 epochs during which it reaches an accuracy of ≈ 92% , 87% , 87% on the train, validation and test sets, respectively.
For analysis, we extract a random test batch of 64 reviews and its states
H ∈ R 64 × 1000 × 256 , where 1000 is the review length when padded with zeros.
One of the main results in (Maheswaranathan et al., 2019) was the observation that the dynamics of the network span a line attractor.
That is, the hidden states of the network are dominantly attracted to a one dimensional manifold, splitting the domain into positive and negative sentiments.
Additionally, Maheswaranathan & Sussillo (2020) study inputs with contextual relations (e.g., the phrase “not bad”), and their effect on the network dynamics.
Our results align with the observations in (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020).
Moreover, we generalize their results by showing that the attracting manifold is in fact of a higher dimension, and that the manifold can be decomposed to semantically understandable components using KANN .
Speciﬁcally, we demonstrate that several Koopman eigenvectors are important in the dynamics, and we can link each of these eigenvectors to a semantically meaningful action.
Thus, in comparison to the framework proposed in (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020), our method naturally decomposes the latent manifold into interpretable attracting components.
In addition, we provide a uniﬁed framework for reasoning and understanding by drawing conclusions directly from the separable building blocks of the latent dynamics.
Most of our results for the sentiment analysis problem are based on the eigendecomposition of C , its resulting eigenvalues { λ j ∈ C } and corresponding eigenvectors { u j ∈ C k } .
For the random states batch H speciﬁed above, we obtain an operator C whose spectrum consists of four eigenvalues with modulus greater than 0 .
99 , i.e., | λ j | > 0 .
In comparison, Maheswaranathan et al.
(2019) identify only a single dominant component.
The values of our λ j read λ 1 = 0 .
λ 2 = 0 .
9965 , and A positive review projected onto { u 1 , u 2 } : A negative review projected onto { u 3 , u 4 } : Two reviews with contextual information projected onto { u 8 } : λ 3 , 4 = 0 .
9942 ±
i 0 .
Consequently, their respective eigenvectors have long memory horizons.
For instance, if we set (cid:15) = 1e − 2 in Eq.
(9), then we get a time stamp τ > 800 for all four eigenvectors.
Namely, these eigenvectors carry information across word sequences of length up to 800 , and only < 2% of the reviews in the IMDB dataset are that long.
In their analysis, the authors of (Maheswaranathan et al., 2019) observe that the network mainly counts positive vs. negative words in a review along a line attractor.
We hypothesize that in our setting, the dominant eigenvectors { u 1 , ..., u 4 } are responsible for this action.
We expect that complex conjugates such as u 3 and u 4 share the same role, e.g., counting the negative words, and we expect that U 12 = { u 1 , u 2 } take the role of counting the positive words.
To verify our hypothesis, we use the readout layer of the model to generate the logits of the state when projected to U 12 and U 34 .
We denote by ˜ y 12 and ˜ y
the logits for ˜ H · V 12 and ˜ H · V 34 , respectively, where V ij denote the i and j columns of V .
For the above test batch, we get perfect correspondence, i.e., ˜ y 12 < . 5 and ˜ y 34 > .
5 on all samples.
In addition to encoding a certain sentiment, the Koopman eigenvectors are advantageous in comparison to a single line attractor as they allow for a direct visualization of the importance of words in a review.
Speciﬁcally, we deﬁne the projection magnitude of a hidden state as follows
[Equation]
We show in Fig.
1 two examples where the magnitude of projection onto U 12 and U 34 clearly highlights positive and negative words, respectively.
In particular, as the network “reads” the review and identiﬁes e.g., a negative word, it increases s ( · ) .
For instance, see mess and muddled in Fig.
Importantly, there may be occurrences of positive/negative n -grams which are not highlighted, such as the word good in the positive example.
We show in App.
A that the above results extend to the entire test set.
Thus, we conclude that { u 1 , ..., u 4 } track positive and negative words.
In addition to { u 1 , ..., u 4 } , we also want to understand how other eigenvectors affect the latent dynamics.
We hypothesize that other vectors are responsible to track contextual information such as ampliﬁers (“extremely good”) and negations (“not bad”).
We collected all reviews that include the phrases “not bad” and “not good” into a single batch, yielding a states tensor with 256 samples.
One way to check our hypothesis is to employ the former visualization using other eigenvectors.
We show two such examples in Fig.
1 in green, where phrases such as “not bad”, “terribly wrong”, and “quite ok” in
In addition to in
4. ECG CLASSIFICATION
Electrocardiogram (ECG) tests track the electrical activity in the heart, and they help detect various abnormalities in a non-invasive way.
Classifying whether a beat is normal or not is a challenging task which lacks descriptive neural models.
A common approach for solving the classiﬁcation problem using neural networks trains an autoencoder model with an L 1 loss over the normal beats.
Classiﬁcation is performed by measuring the loss between the original and reconstructed signals; thus, while it is a classiﬁcation task, ECG classiﬁcation is solved via a regression model.
In particular, high loss values indicate anomalous beats, whereas low values are attributed to normal signals.
Typically, a threshold is set during the training phase, allowing automatic classiﬁcation on the test set.
We ﬁx the threshold to be 26 .
Our network is composed of a single layer LSTM encoder F enc with a hidden size of 64 , and an LSTM decoder F dec with one layer as well.
We use a publicly available subset of the
MIT-BIH arrhythmia database (Goldberger et al., 2000) for our data, named ECG5000 1 .
This dataset includes 5000 sample heartbeats with a sequence length of 140 .
Around 60% of the sequences are classiﬁed as normal and the rest are various anomalous signals.
The model is trained for 150 epochs, yielding an accuracy of 97 .
6% on the train, validation and test sets, respectively.
Similarly to the sentiment analysis problem, we expect the Koopman eigenvectors to take a signiﬁcant role in encoding the latent dynamics.
Given that the network is generative as it is an autoencoder, we hypothesize that the eigenvectors { u j } capture dominant features of normal beats.
Thus, we project normal beats onto pairs of dominant eigenvectors, and decode the resulting hidden states using the decoder to study the obtained signals.
For example, say U 58 , 59 are dominant, then we project onto the space spanned by this pair, then project back and decode.
Using a test batch of 64 normal beats we collect the last hidden state of every sample in a matrix H no ∈ R 64 × 64 and we compute the following.
[Equation]
where | · | is the element-wise modulus of complex numbers.
To determine which eigenvectors are dominant, we employ Eq.
To visualize the results, we take the set of reconstructed signals ¯ X ij of a particular pair ij , and we compute its median µ ij ( t ) for t ∈ [1 , . . . , n ] .
We plot these graphs in Fig.
2 using colors
orange
(pair 58 – 59 ), green (pair 25 – 26 ), red (pair 9 – 10 ), and brown (pair 50 – 51 ).
In addition, the original signals’ median µ (purple) and the median of the signals reconstruction µ r (blue) are shown in each of the subplots for comparison.
Indeed, µ and µ r are almost indistinguishable, and can be differentiated only when zooming in.
Each median graph is wrapped in its median absolute deviation envelope.
We preferred median-based quantities over the common mean and standard deviation since the latter produce cluttered plots in our setting due to outliers.
The plots in Fig.
2 clearly show that each conjugate pair captures a different feature of the time series as marked by the vertical dashed lines.
Speciﬁcally, µ 58 , 59 captures the minimum around t = 3 , and µ 25 , 26 encodes the part of the signal in t ∈ [35 , 75] .
Moreover, µ 9 , 10 attains the maximum at t = 103 , and µ 50 , 51 is approximating the lower peak at t = {+
+}
and we consider these t values to be the salient features.
Importantly, the other Koopman eigenvectors beyond the ones we consider above are less important in the reconstruction, and are mostly helpful in ﬁxing minor variations.
Finally, we provide a similar computation in App.
D using the dominant PCA modes and KernelPCA eigenvectors, where we show that PCA components and KerenelPCA eigenvectors are not useful in identifying the dominant features of beat signals.
Also, we provide a quantitative comparison between the methods.
In addition to identifying principal features of beat signals, we show in what follows that the Koopman eigenvectors are also instrumental in analyzing the latent structure of the LSTM autoencoder.
We begin by showing in Fig.
3 (left) the median values over time of a normal batch and its reconstruction (as in Fig. 2), and similarly for a batch of anomalous signals (orange and brown).
From this data, the task of ECG classiﬁcation may be viewed as a binary classiﬁcation problem, separating normal from anomalous signals via reconstruction.
However, we will now show that this is actually not the case.
Instead, the network essentially encodes inputs, whether normal or anomalous, in a representation that is closer to the manifold of normal signals.
To demonstrate and analyze this phenomenon, we consider hidden state tensors H no
and H an of normal and anomalous beats, respectively, and we concatenate these tensors over the samples yielding H
=
( H no , H an ) ∈ R 128 × 140 × 64 .
We would like to study the decision boundary separating between different signals in the latent space.
To this end, we employ a standard measure known as the silhouette score (Rousseeuw, 1987) to quantify the class separation quality.
The silhouette score σ is a real value in [ − 1 , 1] , where scores close to 1 mean the latent states are well separated.
In contrast, values closer to zero indicate that samples are located on or close to the decision boundary.
We compute the silhouette score estimates on ˜ H
= H · B averaged over samples, and cumulatively averaged over time.
Namely, σ ( t ) = 1 128 · t Σ s,t σ (˜ h s,t ) , where σ (˜ h s,t ) is the silhouette score of the vector ˜ h s,t with s ∈ [1 , . .
, 128] and t ∈ [1 , . . .
We compare three silhouette score estimates denoted by: σ for the original ˜ H , σ PCA for the projection of ˜ H onto its ﬁrst ﬁve principal components, and σ Koopman for the projection of ˜ H onto the ﬁrst ﬁve dominant Koopman eigenvectors.
The results are shown in Fig.
3 (right), where Koopman’s embedding attains low scores compared to PCA and the original states.
Namely, embedding the hidden states using Koopman eigenvectors reveals that the decision boundary between normal and anomalous signals is somewhat blurred, in contrast to the numerical results provided by the reconstructed signals and other embeddings.
This understanding provides a rather straightforward interpretation of the model: it simply encodes the dominant components of all signals as being normal, allowing to easily identify anomalous signals later by measuring their reconstruction error.
Finally, our analysis shows that Koopman eigenvectors successfully identify the salient features of normal beat signals.
We conclude from this observation that the network focuses on identifying these features and reconstructing them accurately.
A correct reconstruction of these salient features allows to subsequently identify using a simple loss check whether a signal is normal or anomalous.
Importantly,
M a t c h i n g p r c
e n t a g e we show in Fig. 3 that the network indeed successfully reconstructs normal and anomalous signals. The understanding that we obtain using KANN is that the network mainly focuses on these salient features during the reconstruction. 4. KANN REPRODUCES THE LATENT DYNAMICS
We now perform a quantitative study of the ability of C to truly capture the latent dynamics.
We will show that indeed, KANN is able to reproduce the nonlinear dynamics of the network in Eq.
(1) to a high degree of precision, and thus we achieve the empirical justiﬁcation to replace F with C .
To this end, we consider the following two metrics:
Relative error of hidden states: let { h s,t } be a collection of states over samples s = 1 , ..., S and across time t = 1 , ..., T .
We generate the predicted collection { h KANN s,t } using Eq.
and we compute
[Equation]
Accuracy error : let G be the neural network component that takes a state and produces the output of the model, i.e., G ( h t ) =
˜
y t .
We denote by ˜ c
t the category predicted by ˜ y t , for instance ˜ c t = arg
max(˜ y t ) .
We compare the difference between ˜ c t and ˜ c KANN t , obtained from G ( h KANN t ) =
˜
y KANN t .
We show in Fig.
4 the results of our quantitative study.
For the sentiment analysis problem (Fig. 4, left), we obtain > % correspondence with the classiﬁcation of the network over all the test set, as is shown for the True Positive (TP) and True Negative (TN) columns vs. the False Positive (FP) and False Negative (FN) columns.
In the ECG classiﬁcation task (Fig. 4, right), we reconstruct 145 signals of the normal test set and compute their loss.
There is a noticeable yet small shift in the loss histogram between the network reconstruction (blue) in comparison to our reconstruction (orange).
However, the threshold for this problem set at 26 during training (black dashed line) yields > 97% agreement in classiﬁcation.
In particular, the false classiﬁcation of normal signals (around loss 90 ) appear both in the network output and in ours.
Finally, we also computed the relative error of the hidden states, obtaining e rel = 0 .
095 on a batch of size 64 for the sentiment analysis task, and e rel = 0 .
0056 on a batch of size 145 for the ECG classiﬁcation problem.
Overall, the above results demonstrate that KANN faithfully represents the latent dynamics.
D ISCUSSION
In this work we presented a novel framework for studying sequence neural models based on Koopman theory and practice.
Our method involves a dimensionality reduction representation of the states, and the computation of a linear map between the current state and the next state.
Key to our approach is the wealth of tools we can exploit from linear analysis and Koopman-related work.
In particular, we compute linear approximations of the state paths via simple matrix-vector multiplications.
Moreover, we identify dominant features of the dynamical system and their effect on inference and prediction.
Our results on the sentiment analysis problem, and the ECG classiﬁcation challenge provide simple yet accurate descriptions of the underlying dynamics and behavior of the recurrent models.
Our work lays the foundations to further develop application-based descriptive frameworks, towards an improved understanding of neural networks.
In the future, we plan to explore our framework during the training of the model, where in this work we focused only on trained models.