S EQ 2T ENS : A N E FFICIENT R EPRESENTATION OF S EQUENCES BY L OW -R ANK T ENSOR P ROJECTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies.
At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning.
We use a classical mathematical object – the free algebra – to capture this non-commutativity.
To address the innate computational complexity of this algebra, we use compositions of low-rank tensor projections.
This yields modular and scalable building blocks that give state-of-the-art performance on standard benchmarks such as multivariate time series classiﬁcation and generative models for video.
I NTRODUCTION
A central task of learning is to ﬁnd representations of the underlying data that efﬁciently and faithfully capture their structure.
In the case of sequential data, one data point consists of a sequence of objects.
This is a rich and non-homogeneous class of data and includes classical uni- or multi-variate time series (sequences of scalars or vectors), video (sequences of images), and text (sequences of letters).
Particular challenges of sequential data are that each sequence entry can itself be a highly structured object and that data sets typically include sequences of different length which makes naive vectorization troublesome.
Contribution.
Our main result is a generic method that takes a static feature map for a class of objects (e.g. a feature map for vectors, images, or letters) as input and turns this into a feature map for sequences of arbitrary length of such objects (e.g. a feature map for time series, video, or text).
We call this feature map for sequences Seq2Tens for reasons that will become clear; among its attractive properties are that it (i) provides a structured, parsimonious description of sequences; generalizing classical methods for strings, (ii) comes with theoretical guarantees such as universality, (iii) can be turned into modular and ﬂexible neural network (NN) layers for sequence data.
The key ingredient to our approach is to embed the feature space of the static feature map into a larger linear space that forms an algebra (a vector space equipped with a multiplication).
The product in this algebra is then used to “stitch together” the static features of the individual sequence entries in a structured way.
The construction that allows to do all this is classical in mathematics, and known as the free algebra (over the static feature space).
Outline.
Section 2 formalizes the main ideas of Seq2Tens and introduces the free algebra T( V ) over a space V as well as the associated product, the so-called convolution tensor product .
It shows how low rank (LR) constructions combined with sequence-to-sequence transforms allows one to efﬁciently use this rich algebraic structure.
Section 3 applies the results of Section 2 to build modular and scalable NN layers.
Section 4 demonstrates the ﬂexibility and modularity of this approach on both discriminative and generative benchmarks.
Section 6 makes connections with previous work and summarizes this article.
In the appendices we provide mathematical background, extensions, and detailed proofs for our theoretical results.
C APTURING ORDER BY NON - COMMUTATIVE MULTIPLICATION
We denote the set of sequences of elements in a set X by
[Equation]
where L ≥ 1 is some arbitrary length.
Even if X itself is a linear space, e.g. X =
R , Seq( X ) is never a linear space since there is no natural addition of two sequences of different length.
Seq2Tens in a nutshell.
Given any vector space V we may construct the so-called free algebra
T( V ) over V .
We describe the space T( V ) in detail below, but as for now the only thing that is important is that T( V ) is also a vector space that includes V , and that it carries a non-commutative product, which is, in a precise sense, “the most general product” on V .
The main idea of Seq2Tens is that any “static feature map” for elements in X
[Equation]
can be used to construct a new feature map Φ : Seq( X ) → T( V ) for sequences in X by using the algebraic structure of T( V ) : the non-commutative product on T( V ) makes it possible to “stitch together” the individual features φ ( x 1 ) , . .
, φ ( x L ) ∈ V ⊂ T( V ) of the sequence x in the larger space
T( V ) by multiplication in T( V ) .
With this we may deﬁne the feature map Φ( x ) for a sequences x = ( x 1 , . . .
, x L )
∈ Seq( X ) as follows
(i) lift the map φ : X → V to a map ϕ : X → T( V ) , (ii) map Seq( X ) → Seq(T( V )) by ( x 1 , . . .
, x L ) (cid:55)→ ( ϕ ( x 1 ) , . . .
, ϕ ( x L )) , (iii) map Seq(T( V ))
→ T( V ) by multiplication ( ϕ ( x 1 ) , . . .
ϕ ( x L ))
(cid:55)→ ϕ ( x 1 ) · · · ϕ ( x L ) .
In a more concise form, we deﬁne Φ as
[Equation]
where (cid:81) denotes multiplication in T( V ) .
We refer to the resulting map Φ as the Seq2Tens map, which stands short for Seq uences-
2 - Tens ors .
Why is this construction a good idea?
First note, that step (i) is always possible since V ⊂ T( V ) and we discuss the simplest such lift before Theorem 2.
as well as
other choices in Appendix B. Further, if φ , respectively ϕ , provides a faithful representation of objects in X , then there is no loss of information in step (ii).
Finally, since step (iii) uses “the most general product” to multiply ϕ ( x 1 ) · · · ϕ ( x L ) one expects that Φ( x ) ∈ T( V ) faithfully represents the sequence x as an element of T( V ) .
Indeed in Theorem 2.1 below we show an even stronger statement, namely that if the static feature map φ : X → V contains enough non-linearities so that non-linear functions from X to R can be approximated as linear functions of the static feature map φ , then the above construction extends this property to functions of sequences.
Put differently, if φ is a universal feature map for X , then Φ is a universal feature map for Seq( X ) ; that is, any non-linear function f ( x ) of a sequence x can be approximated as a linear functional of Φ( x ) , f ( x ) ≈ (cid:104) (cid:96), Φ( x ) (cid:105) .
We also emphasize that the domain of Φ is the space Seq( X ) of sequences of arbitrary (ﬁnite) length.
The remainder of this Section gives more details about steps (i),(ii),(iii) for the construction of Φ .
The free algebra T( V ) over a vector space V .
Let V be a vector space.
We denote by T( V ) the set of sequences of tensors indexed by their degree m ,
[Equation]
where by convention V ⊗
= R .
For example, if V = R d and t = ( t m ) m ≥ 0 is some element of
T( R d ) , then its degree m = 1 component is a d -dimensional vector t 1 , its degree m = 2 component is a d × d matrix t 2 , and its degree m = 3 component is a degree 3 tensor t 3 .
By deﬁning addition and scalar multiplication as
[Equation]
the set T( V ) becomes a linear space.
By identifying v ∈ V as the element (0 , v, 0 , . . . , 0) ∈ T( V ) we see that V is a linear subspace of T( V ) .
Moreover, while V is only a linear space, T( V ) carries a product that turns T( V ) into an algebra.
This product is the so-called tensor convolution product , and is deﬁned for s , t ∈ T( V ) as
[Equation]
where ⊗ denotes the usual outer tensor product; e.g. for vectors
u
= ( u i ) , v = ( v i ) ∈
R d the outer tensor product u ⊗ v is the d × d matrix ( u i v j ) i,j =1 ,...,d .
We emphasize that like the outer tensor product ⊗ , the tensor convolution product · is non-commutative, i.e. s · t (cid:54) =
t · s .
In a mathematically precise sense, T( V ) is the most general algebra that contains V ; it is a “free construction” .
Since
T( V ) is realized as series of tensors of increasing degree, the free algebra T( V ) is also known as the tensor algebra in the literature.
Appendix A contains background on tensors and further examples.
Lifting static feature maps.
Step (i) in the construction of Φ requires turning a given feature map φ : X → V into a map ϕ : X → T( V ) .
Throughout the rest of this article we use the lift
[Equation]
We discuss other choices in Appendix B, but attractive properties of the lift 6 are that (a) the evaluation of Φ against low rank tensors becomes a simple recursive formula (Proposition 3.3, (b) it is a generalization of sequence sub-pattern matching as used in string kernels (Appendix B.3, (c) despite its simplicity it performs exceedingly well in practice (Section 4).
Extending to sequences of arbitrary length.
So far we have speciﬁed in steps (i) and (ii) how the map Φ : X → T( V ) behaves on single observations.
Step (iii) in the construction amounts to the requirement that for any two sequences x = ( x 1 , . . .
, x K ) ,
y = ( y 1 , . . .
, y L )
∈
Seq( V ) , their concatenation deﬁned as z = ( x 1 , . . .
, x K , y 1 , . . .
, y L )
∈
Seq( V ) can be understood in the feature space as (non-commutative) multiplication of their corresponding features
[Equation]
Using equation 7 we may inductively extend the lift ϕ to sequences of arbitrary length by starting from sequences consisting of a single observation, which is equivalent to equation 2.
By repeatedly using the deﬁnition of the tensor convolution product in equation 5 this may also be written as
[Equation]
where x = ( x 1 , . .
, x L )
∈
Seq( V ) and the summation is over non-contiguous subsequences of x .
Some intuition: generalized pattern matching.
Our derivation of the feature map Φ( x ) =
(1 , Φ 1 ( x ) , Φ 2 ( x ) , . . . )
∈ T( V ) was guided by general algebraic principles, but equation 8 provides an intuitive interpretation.
It shows that for each m ≥ 1 , the entry Φ m ( x ) ∈ V ⊗ m constructs a summary of a long sequence x = ( x 1 , . . .
, x L )
∈
Seq( V ) based on subsequences ( x i 1 , . .
, x i m ) of x of length m .
It does this by taking the usual outer tensor product x i 1 ⊗ · · · ⊗ x i m ∈ V ⊗ m and summing over all possible subsequences.
This is completely analogous to how string kernels provide a structured description of text by looking at non-contiguous substrings of length m (indeed, Appendix B.3 makes this rigorous).
However, the main difference is that the above construction works for arbitrary sequences and not just sequences of discrete letters.
Readers with less mathematical background might simply take this as motivation and regard equation 8 as deﬁnition.
However, the algebraic background allows to prove that Φ is universal, see Theorem 2.1 below.
Universality.
A function φ : X → V is said to be universal for X if all continuous functions on
X can be approximated as linear functions on the image of φ .
One of the most powerful features of neural nets is their universality (Hornik, 1991).
A very attractive property of Φ is that it preserves universality: if φ : X → V is universal for X , then Φ : Seq( X ) → T( V ) is universal for Seq( X ) .
To make this precise, note that V ⊗ m is a linear space and therefore any (cid:96) =
( (cid:96) 0 , (cid:96) 1 , . . .
, (cid:96) M , 0 , 0 , . . . ) ∈
T( V ) consisting of M tensors (cid:96) m ∈ V ⊗ m , yields a linear functional on T( V ) ;
e.g. if V = R d and we identify (cid:96) m in coordinates as (cid:96) m =
( (cid:96) i 1 ,...,i m m )
i 1 ,...,i m ∈{ 1 ,...,d } then (cid:104) (cid:96), t (cid:105) :
=
M (cid:88) m = (cid:104) (cid:96) m , t m (cid:105) =
M (cid:88) m
= (cid:88)
i 1 ,...,i m ∈{ 1 ,...,d } (cid:96) i 1 ,...,i m m t i 1 ,...,i m m .
Thus linear functionals of the feature map Φ , are real-valued functions of sequences.
Theorem 2.
below shows that any continuous function f : Seq( X ) → R can by arbitrary well approximated by a (cid:96) ∈ T( V ) , f ( x ) ≈ (cid:104) (cid:96), Φ( x ) (cid:105) .
Theorem 2.1.
Let φ : X → V be a universal map with a lift that satisﬁes some mild constraints, then the following map is universal:
[Equation]
A detailed proof and the precise statement of Theorem 2.1 is given in Appendix B.
A PPROXIMATION BY LOW - RANK LINEAR FUNCTIONALS
The combinatorial explosion of tensor coordinates and what to do about it.
The universality of Φ suggests the following approach to represent a function f : Seq( X )
→ R of sequences:
First compute Φ( x ) and then optimize over (cid:96) (and possibly also the hyperparameters of φ ) such that f ( x ) ≈ (cid:104) (cid:96), Φ( x ) (cid:105) =
(cid:80)
Mm =
(cid:104) (cid:96) m ,
Φ m ( x ) (cid:105) .
Unfortunately, tensors suffer from a combinatorial explosion in complexity in the sense that even just storing Φ m ( x ) ∈ V ⊗ m ⊂ T( V ) requires O (dim( V ) m ) real numbers.
Below we resolve this computational bottleneck as follows: in Proposition 3.3 we show that for a special class of low-rank elements (cid:96) ∈ T( V ) , the functional x (cid:55)→ (cid:104) (cid:96), Φ( x ) (cid:105) can be efﬁciently computed in both time and memory.
This is somewhat analogous to a kernel trick since it shows that (cid:104) (cid:96), Φ( x ) (cid:105) can be cheaply computed without explicitly computing the feature map Φ( x ) .
However, Theorem 2.1 guarantees universality under no restriction on (cid:96) , thus restriction to rank- 1 functionals limits the class of functions f ( x ) that can be approximated.
Nevertheless, by iterating these “low-rank functional” constructions in the form of sequence-to-sequence transformations this can be ameliorated.
We give the details below but to gain intuition, we invite the reader to think of this iteration analogous to stacking layers in neural network: each layer is a relatively simple non-linearity (e.g. a sigmoid composed with an afﬁne function) but by composing such layers, complicated functions can be efﬁciently approximated.
Rank-1 functionals are computationally cheap.
Degree m =
2 tensors are matrices and LR approximations of matrices are widely used in practice (Udell & Townsend, 2019) to address the quadratic complexity.
The deﬁnition below generalizes the rank of matrices (tensors of degree m = 2 ) to tensors of any degree m .
Deﬁnition 3.1.
The rank (also called CP rank (Carroll & Chang, 1970)) of a degree m tensor t m ∈ V ⊗ m is the smallest number r ≥ 0 such that one may write
[Equation]
We say that t =
( t m ) m ≥ 0 ∈ T( V )
has rank-1 (and degree M ) if each t m ∈ V ⊗ m is a rank-1 tensor and t i =
for i > M .
Remark 3.2.
For x = ( x 1 , . . .
, x L )
∈
Seq( V ) , the rank r m ∈ N of Φ m ( x ) satisﬁes r m ≤ (cid:0) Lm (cid:1) , while the rank and degree r, d ∈ N of Φ( x ) satisfy r ≤ (cid:0) LK (cid:1) for K = (cid:4) L 2 (cid:5) and d ≤ L .
A direct calculation shows that if (cid:96) is of rank- 1 , then (cid:104) (cid:96), Φ( x ) (cid:105) can be computed very efﬁciently by inner product evaluations in V .
Proposition 3.3.
Let (cid:96) =
( (cid:96) m )
m
≥ 0 ∈ T( V ) be of rank-1 and degree M .
If φ is lifted to ϕ as in equation 6, then (cid:104) (cid:96),
[Equation]
=
[Equation]
(12) where (cid:96) m
= v m 1 ⊗ · · · ⊗ v mm ∈ V ⊗ m , v m
i ∈ V and m = 0 , . .
, M .
Note that the inner sum is taken over all non-contiguous subsequences of x of length m , analogous to m -mers of strings and we make this connection precise in Appendix B.3; the proof of Proposition 3.3 is given in Appendix B.1.1.
While equation 12 looks expensive, by casting it into a recursive formulation over time, it can be computed in O ( M 2 · L · d ) time and O ( M 2 · ( L + c )) memory, where d is the inner product evaluation time on V , while c is the memory footprint of a v ∈ V . This can further be reduced to O ( M · L · d ) time and O ( M · ( L + c )) memory by an efﬁcient parametrization of the rank- 1 element (cid:96) ∈ T( V ) . We give further details in Appendices D.2, D.3, D.4.
Low-rank Seq2Tens maps.
The composition of a linear map L : T( V ) → R N with Φ can be computed cheaply in parallel using equation 12 when L is speciﬁed through a collection of N ∈ N rank-1 elements (cid:96) 1 , . . .
, (cid:96) N ∈ T( V ) such that
[Equation]
We call the resulting map ˜Φ ˜ θ : Seq( X ) → R N a L ow-rank S eq 2T ens map of width- N and order- M , where M ∈ N is the maximal degree of (cid:96) 1 , . . .
, (cid:96) N such that (cid:96) ji = 0 for i > M .
The LS2T map is parametrized by (1) the component vectors v kj,m ∈ V of the rank- 1 elements (cid:96) jm = v 1 j,m ⊗· · ·⊗ v mj,m , (2) by any parameters θ that the static feature map φ θ : X → V may depend on.
We jointly denote these parameters by ˜ θ
= ( θ, (cid:96) 1 , . . .
, (cid:96) N ) .
In addition, by the subsequent composition of ˜Φ ˜ θ with a linear functional R N → R , we get the following function subspace as hypothesis class for the LS2T
[Equation]
Hence, we acquire an intuitive explanation of the (hyper)parameters: the width of the LS2T, N ∈ N speciﬁes the maximal rank of the low-rank linear functionals of Φ that the LS2T can represent, while the span of the rank- 1 elements, span( (cid:96) 1 , . . .
, (cid:96) N ) determine an N -dimensional subspace of the dual space of T( V ) consisting of at most rank- N functionals.
Recall now that without rank restrictions on the linear functionals of Seq2Tens features, Theorem 2.1 would guarantee that any real-valued function f : Seq( X ) → R could be approximated by f ( x ) ≈ (cid:104) (cid:96), Φ( x 1 , . . .
, x L ) (cid:105) .
As pointed out before, the restriction of the hypothesis class to lowrank linear functionals of Φ( x 1 , . .
, x L ) would limit the class of functions of sequences that can be approximated.
To ameliorate this, we propose to use LS2T transforms in a sequence-to-sequence fashion that allows us to stack such low-rank functionals, signiﬁcantly recovering expressiveness.
Sequence-to-sequence transforms.
We can use LS2T to build sequence-to-sequence transformations in the following way: ﬁx the static map φ θ : X → V parametrized by θ and rank-
1 elements such that ˜ θ
= ( θ, (cid:96) 1 , . . .
, (cid:96) N ) and apply the resulting LS2T map ˜Φ ˜ θ over expanding windows of x :
[Equation]
The cost of computing the expanding window sequence-to-sequence transform in equation 15 is no more expensive than computing ˜Φ ˜ θ ( x 1 , . . .
, x L ) itself due to the recursive nature of our algorithms.
Stacking sequence-to-sequence transforms.
Inspired by the empirical successes of stacked
RNNs (Graves et al., 2013b;a; Sutskever et al., 2014), we iterate the transformation 15 D -times:
[Equation]
Each of these mappings Seq( R N i )
→ Seq( R N i +1 ) is parametrized by the parameters ˜
θ i of a static feature map φ θ i and a linear map L i speciﬁed by N i rank-1 elements of T( V ) ; these parameters are collectively denoted by ˜ θ i
=
( θ i , (cid:96) 1 i , . . .
, (cid:96) N i i ) .
Evaluating the ﬁnal sequence in Seq( R N D ) at last observation-time t
=
L , we get the D-times stacked LS2T map
[Equation]
Making precise how the stacking of such low-rank sequence-to-sequence transformations approximates general functions requires more tools from algebra, and we provide a rigorous quantitative statement in Appendix C. Here, we just appeal to the analogy made with adding depth in neural networks mentioned earlier and empirically validate this in our experiments in Section 4.
I npu t (cid:96) d
T i m e /
C onv ( 8 )
B N / A c ti v a ti on
T i m e /
C onv ( 5 )
B N / A c ti v a ti on
T i m e /
C onv ( 3 )
B N / A c ti v a ti on
T i m e /
D i ff e r e n ce
L S 2 T 64 ×
B N / F l a tt e n
T i m e /
D i ff e r e n ce
L S 2 T 64 ×
B N / F l a tt e n
T i m e /
D i ff e r e n ce
LS2T 64 ×
BN/Flatten
FCNblock
StackedLS2Tblock
B UILDING NEURAL NETWORKS WITH LS2T LAYERS
The Seq2Tens map Φ built from a static feature map φ is universal if φ is universal, Theorem 2.1.
NNs form a ﬂexible class of universal feature maps with strong empirical success for data in X = R d , and thus make a natural choice for φ .
Combined with standard deep learning constructions, the framework of Sections 2 and 3 can build modular and expressive layers for sequence learning.
Neural LS2T layers.
The simplest choice among many is to use as static feature map φ :
X = R d → R M a feedforward network with depth- P , φ = φ P ◦ · · · ◦ φ 1 where φ j ( x ) = σ ( W j x + b j ) for W j ∈ R M × d , b j ∈ R M .
We can then lift this to a map ϕ : R d → T( R M ) as prescribed in equation 6.
Hence, the resulting LS2T layer x
(cid:55)→ (˜Φ ˜ θ ( x 1 , . . .
, x i ))
i =1 ,...,L is a sequence-to-sequence transform Seq( R d ) →
Seq( R M ) that is parametrized by ˜ θ =
( W 1 , b 1 , . . .
, W P , b P , (cid:96) 11 , . . .
, (cid:96) N 1 1 ) .
Bidirectional LS2T layers.
The transformation in equation 15 is completely causal in the sense that each step of the output sequence depends only on past information.
For generative models, it can behove us to make the output depend on both past and future information, see Graves et al.
(2013a);
Baldi et al.
; Li & Mandt (2018).
Similarly to bidirectional RNNs and LSTMs (Schuster &
Paliwal, 1997; Graves & Schmidhuber, 2005), we may achieve this by deﬁning a bidirectional layer, ˜Φ b(˜ θ 1 , ˜ θ 2 ) ( x ) :
Seq( R d ) →
Seq( R N + N
[Equation] [Equation]
The sequential nature is kept intact by making the distinction between what classiﬁes as past (the ﬁrst N coordinates) and future (the last N (cid:48) coordinates) information.
This amounts to having a form of precognition in the model, and has been applied in e.g. dynamics generation (Li & Mandt, 2018), machine translation (Sundermeyer et al., 2014), and speech processing (Graves et al., 2013a).
Convolutions and LS2T. We motivate to replace the time-distributed feedforward layers proposed in the paragraph above by temporal convolutions (CNN) instead.
Although theory only requires the preprocessing layer of the LS2T to be a state-space nonlinearity, we ﬁnd that it is beneﬁcial to capture some of the sequential information in the preprocessing layer as well, e.g. using CNNs or RNNs.
From a mathematical point of view, CNNs are a straightforward extension since they can be interpreted as time-distributed feedforward layers applied to the input sequence augmented with a p ∈ N number of its lags for CNN kernel size p (see Appendix D.1 for further discussion).
In the following, we precede our stacked LS2T blocks by one or more CNN layers.
Intuitively, CNNs and LS2Ts are similar in that both transformations operate on subsequences of their input sequence.
The main difference between the two lies in that CNNs operate on contiguous subsequences , and therefore, capture local, short-range nonlinear interactions between timesteps; while LS2Ts (equation 12) use all non-contiguous subsequences , and hence, learn global, long-range interactions in time.
This observation motivates that the inductive biases of the two types of layers (local/global time-interactions) are highly complementary in nature, and we propose that the improvement in the experiments on the models containing vanilla CNN blocks are due to this complementarity.
E XPERIMENTS
We demonstrate the modularity and ﬂexibility of the above LS2T and its variants by applying it to (i) multivariate time series classiﬁcation, (ii) mortality prediction in healthcare, (iii) generative modelling of sequential data.
In all cases, we take a strong baseline model (FCN and GP-VAE, as detailed below) and upgrade it with LS2T layers.
As Thm.
2.1 requires the Seq2Tens to be preceded by at
[Table]
least a universal state-space nonlinearity, we expect these layers to perform best as an add-on on top of other models, which however can be quite simple, such as an FCN.
The additional computation time is negligible (in fact, for FCN it allows us to reduce the number of parameters signiﬁcantly), but it can yield substantial improvements.
This is remarkable, since the results of the original models are already state-of-the-art on well-established (frequentist and Bayesian) benchmarks.
5. M ULTIVARIATE TIME SERIES CLASSIFICATION
First, we consider multivariate time series classiﬁcation (TSC) on a semi-standardized collection of benchmark datasets uniﬁed in Baydogan (2015).
A wide range of previous publications report results on this archive, which makes it possible to compare against a wide range of baseline methods without bias in parameter settings.
A subset of this archive was also considered in a recent, highly cited review paper on deep learning (DL) for TSC (Ismail Fawaz et al., 2019) from where we also borrow the best performing models as baselines.
We provide further details on the experiment, baselines and datasets in Appendix E.1.
As our own models, we introduce two simple architectures that utilize
LS2T layers: (i) LS2T 3 stacks 3 LS2T layers of order 2 and width 64 ; (ii) FCN-LS2T 3 precedes the LS2T 3 block by a CNN block of 3 convolutional layers of width 64 and ﬁlter sizes of 8 , 5 , 3 ; a downsized version of the FCN model of (Wang et al., 2017) with time-embeddings (Liu et al., 2018a) and without a ﬁnal global average pooling (GAP) layer, see Fig.
The hyperparameters, i.e. CNN/LS2T width N = 64 , LS2T order M = 2 and LS2T depth D = 3 were tuned using a small subset of the datasets from N ∈ { 16 , 64 , 128 } , M ∈ { 2 , 3 , 4 } and D ∈ { 1 , 2 , 3 } .
Results.
We trained the introduced models on each dataset 5 times while the baseline results were borrowed from their respective publications.
In Appendix E.1, Figure 4 depicts the results as a boxplot of distributions and a CD diagram using the Nemenyi test (Nemenyi, 1963), while Table 7 shows the full list of means and standard deviations.
Since mean-ranks based tests raise some paradoxical issues (Benavoli et al., 2016), it is customary to conduct pairwise comparisons using frequentist (Demˇsar, 2006) or Bayesian (Benavoli et al., 2017) hypothesis tests.
We adopted the Bayesian signed-rank test from Benavoli et al.
(2014), which is showcased in Table 1, while the Bayesian posteriors are visualized on Figure 5 in App.
E.1.
The results of the signed-rank test indicate that while LS2T 3 is better with moderate probability ( p ≥ 0 . 6) than 4 of the baselines, FCN-LS2Tperforms better than all with high probability ( p ≥ 0 . 75 ), except for MLSTMFCN which is only slightly outperformed by FCN-LS2T 3 .
Note however that FCN-LS2T 3 has fewer parameters than
MLSTMFCN by more than 60% , see Table 6 for a parameter comparison.
Also, as MLSTMFCN is the concatenation of an FCN with an LSTM (Karim et al., 2019), the LSTM layer can bottleneck the computations for very long sequences as it scales linearly, while LS2T layers effectively scale sublinearly in sequence length , Appendix D.4 for a computational comparison.
In conclusion, we have veriﬁed that LS2T 3 already performs well on some classiﬁcation tasks, and by preceding it with an FCN, its performance is elevated to outperforming all baseline models.
This observation is not surprising, as Thm. 2.1 warrants, the full strength of the LS2T layer shines when it is preceded by at least some state-space nonlinearities.
Hence, we expect LS2Ts to ﬁnd their best use as a modular building block being part of larger models.
We also suggest the improvement on the vanilla FCN composed of CNN blocks is due to the LS2T enhancing the model’s ability to pick up long-range interactions in time as previously discussed at the end of Section 4.
5. M ORTALITY PREDICTION
We consider the 2012 PhysioNet challenge dataset (Goldberger et al., 2000) for mortality prediction, that consists of medical time series of 12,000 ICU stays over at least 48 hours.
Overall, 6 static features were recorded and potentially up to 37 TS values were measured both irregularly and asynchronously for each stay with certain dimensions completely missing in some cases.
The task is to predict whether a patient died during their hospital stay.
This is a trickier dataset due to missingness, low signal-to-noise ratio, and unbalanced class distributions with a prevalence ratio of around 14% .
We extend the experiments conducted in Horn et al.
(2020) containing very strong baselines, and under the same experimental setting we train on this dataset our model FCN-LS2T 3 and the FCN as control.
We preprocess the data using the method introduced in Che et al.
(2018, eq.
(9)), which includes adding the observation mask and time since last measurement taken as extra TS dimensions, while we handle static features by tiling them along the time axis and also adding them as extra dims.
For the models FCN-LS2T 3 and FCN, the same hyperparameters are used as in the TSC experiment with the additional introduction of a SpatialDropout1D layer following each CNN and LS2T layer using a dropout rate of r = 0 .
1 in order to deal with the low SNR of the dataset.
[Table]
Results.
Table 2 compares the results of our model, FCN-LS2T 3 with that of FCN and the baselines from Horn et al.
(2020) through 3 metrics: (1) classiﬁcation accuracy, (2) area under the precision-recall curve (AUPRC), (3) area under the ROC curve (AUROC).
We observe that FCNLS2T 3 signiﬁcantly outperforms FCN according to all metrics.
FCN-LS2T 3 is also competitive with the SOTA methods, such as the T RANSFORMER (Vaswani et al., 2017), S E FT (Horn et al., 2020), IP-N ETS (Shukla & Marlin, 2019) and GRU-D (Che et al., 2018), achieving the best accuracy and taking fourth place with respect to both AUPRC and AUROC.
5. G ENERATING SEQUENTIAL DATA
I npu t (cid:96) d
P r e p r o ce ss o r d
C onvo l u ti on h
T i m e /
D i ff e r e n ce h
+
B - L S 2 T h ×
L N / F l a tt e n 4 h
D e n s e h (cid:48) ... D e n s e h (cid:48)
L a t e n t 3 d (cid:48)
As another benchmarking experiment, we demonstrate that LS2T layers do not only provide good representations of sequences in discriminative, but also generative models.
The considered problem is sequential data imputation for time series and video.
The GP-VAE model.
In this experiment, we take as base model the recent GP-VAE (Fortuin et al., 2020), that provides state-of-the-art results for probabilistic sequential data imputation.
The
GP-VAE is essentially based on the HI-VAE (Nazabal et al., 2018) for handling missing data in variational autoencoders (VAEs) (Kingma & Welling, 2013) adapted to the handling of time series data by the use of a Gaussian process (GP) prior (Williams & Rasmussen, 2006) across time in the latent sequence space to capture temporal dynamics.
Since the GP-VAE is a highly advanced model, its in-depth description is deferred to Appendix E.2.We extend the experiments conducted in Fortuin et al.
and we make one simple change to the GP-VAE architecture without changing any other hyperparameters or aspects: we introduce a single bidirectional LS2T layer (B-LS2T) into the encoder network that is used in the amortized representation of the means and covariances of the variational posterior.
The B-LS2T layer is preceded by a time-embedding and differencing block, and succeeded by channel ﬂattening and layer normalization as depicted in Figure 2.
The idea behind this experiment is to see if we can improve the performance of a highly complicated model that is composed of many interacting submodels, by the naive introduction of LS2T layers.
[Table]
Results.
To make the comparison, we ceteris paribus re-ran all experiments the authors originally included in their paper (Fortuin et al., 2020), which are imputation of Healing MNIST, Sprites, and Physionet 2012.
The results are in Table 3, which report the same metrics as used in Fortuin et al.
, i
[Table]
e. negative log-likelihood (NLL, lower is better), mean squared error (MSE, lower is better) on test sets
[Table]
and downstream classiﬁcation performance of a linear classiﬁer (AUROC, higher is better).
For all other models beside our GP-VAE (B-LS2T), the results were borrowed from Fortuin et al.
We observe that simply adding the B-LS2T layer improved the result in almost all cases, except for Sprites, where the GP-VAE already achieved a very low MSE score.
Additionally, when comparing GP-VAE to BRITS on Physionet, the authors argue that although the
BRITS achieves a higher AUROC score, the GP-VAE should not be disregarded as it ﬁts a generative model to the data that enjoys the usual Bayesian beneﬁts of predicting distributions instead of point predictions.
The results display that by simply adding our layer into the architecture, we managed to elevate the performance of GP-VAE to the same level while retaining these same beneﬁts.
We believe the reason for the improvement is a tighter amortization gap in the variational approximation (Cremer et al., 2018) achieved by increasing the expressiveness of the encoder by the LS2T allowing it to pick up on long-range interactions in time.
We provide further discussion in Appendix E.2.
R ELATED WORK AND S UMMARY
Related Work.
The literature on tensor models in ML is vast.
Related to our approach we mention pars-pro-toto Tensor Networks (Cichocki et al., 2016), that use classical LR decompositions, such as CP (Carroll & Chang, 1970), Tucker (Tucker, 1966), tensor trains (Oseledets, 2011) and tensor rings (Zhao et al., 2019); further, CNNs have been combined with LR tensor techniques (Cohen et al., 2016; Kossaiﬁ et al., 2017) and extended to RNNs (Khrulkov et al., 2019); Tensor Fusion
Networks (Zadeh et al., 2017) and its LR variants (Liu et al., 2018b; Liang et al., 2019; Hou et al., 2019); tensor-based gait recognition (Tao et al., 2007).
Our main contribution to this literature is the use of the free algebra T( V ) with its convolution product · , instead of V ⊗ m with the outer product ⊗ that is used in the above papers.
While counter-intuitive to work in a larger space T( V ) , the additional algebra structure of (T( V ) , · ) is the main reason for the nice properties of Φ ( universality, making sequences of arbitrary length comparable, convergence in the continuous time limit ; see
Appendix B) which we believe are in turn the main reason for the strong benchmark performance .
Stacked LR sequence transforms allow to exploit this rich algebraic structure with little computational overhead.
Another related literature are path signatures in ML (Lyons, 2014;
Chevyrev &
Kormilitzin, 2016; Graham, 2013; Bonnier et al., 2019; Toth & Oberhauser, 2020).
These arise as special case of Seq2Tens (Appendix B) and our main contribution to this literature is that Seq2Tens resolves a well-known computational bottleneck in this literature since it never needs to compute and store a signature , instead it directly and efﬁciently learns the functional of the signature .
Summary.
We used a classical non-commutative structure to construct a feature map for sequences of arbitrary length.
By stacking sequence transforms we turned this into scalable and modular NN layers for sequence data.
The main novelty is the use of the free algebra T( V ) constructed from the static feature space V .
While free algebras are classical in mathematics, their use in ML seems novel and underexplored.
We would like to re-emphasize that (T( V ) , · ) is not a mysterious abstract space: if you know the outer tensor product ⊗ then you can easily switch to the tensor convolution product · by taking sums of outer tensor products, as deﬁned in equation 5.
As our experiments show, the beneﬁts of this algebraic structure are not just theoretical but can signiﬁcantly elevate performance of already strong-performing models.