S EQ 2T ENS : A N E FFICIENT R EPRESENTATION OF S EQUENCES BY L OW -R ANK T ENSOR P ROJECTIONS
A BSTRACT
Sequential data such as time series, video, or text can be challenging to analyse as the ordered structure gives rise to complex dependencies.
At the heart of this is non-commutativity, in the sense that reordering the elements of a sequence can completely change its meaning.
We use a classical mathematical object – the free algebra – to capture this non-commutativity.
To address the innate computational complexity of this algebra, we use compositions of low-rank tensor projections.
This yields modular and scalable building blocks that give state-of-the-art performance on standard benchmarks such as multivariate time series classiﬁcation and generative models for video.
I NTRODUCTION
A central task of learning is to ﬁnd representations of the underlying data that efﬁciently and faithfully capture their structure.
In the case of sequential data, one data point consists of a sequence of objects.
This is a rich and non-homogeneous class of data and includes classical uni- or multi-variate time series (sequences of scalars or vectors), video (sequences of images), and text (sequences of letters).
Particular challenges of sequential data are that each sequence entry can itself be a highly structured object and that data sets typically include sequences of different length which makes naive vectorization troublesome.
Contribution.
Our main result is a generic method that takes a static feature map for a class of objects (e.g. a feature map for vectors, images, or letters) as input and turns this into a feature map for sequences of arbitrary length of such objects (e.g. a feature map for time series, video, or text).
We call this feature map for sequences Seq2Tens for reasons that will become clear; among its attractive properties are that it (i) provides a structured, parsimonious description of sequences; generalizing classical methods for strings, (ii) comes with theoretical guarantees such as universality, (iii) can be turned into modular and ﬂexible neural network (NN) layers for sequence data.
The key ingredient to our approach is to embed the feature space of the static feature map into a larger linear space that forms an algebra (a vector space equipped with a multiplication).
The product in this algebra is then used to “stitch together” the static features of the individual sequence entries in a structured way.
The construction that allows to do all this is classical in mathematics, and known as the free algebra (over the static feature space).
Outline.
Section 2 formalizes the main ideas of Seq2Tens and introduces the free algebra T( V ) over a space V as well as the associated product, the so-called convolution tensor product .
Section 3 shows how low rank (LR) constructions combined with sequence-to-sequence transforms allows one to efﬁciently use this rich algebraic structure.
Section 4 applies the results of Sections 2 and 3 to build modular and scalable NN layers for sequential data.
Section 5 demonstrates the ﬂexibility and modularity of this approach on both discriminative and generative benchmarks.
Sectionmakes connections with previous work and summarizes this article.
In the appendices we provide mathematical background, extensions, and detailed proofs for our theoretical results.
C APTURING ORDER BY NON - COMMUTATIVE MULTIPLICATION
We denote the set of sequences of elements in a set X by
[Equation]
where L ≥ 1 is some arbitrary length.
Even if X itself is a linear space, e.g. X =
R , Seq( X ) is never a linear space since there is no natural addition of two sequences of different length.
Seq2Tens in a nutshell.
Given any vector space V we may construct the so-called free algebra
T( V ) over V .
We describe the space T( V ) in detail below, but as for now the only thing that is important is that T( V ) is also a vector space that includes V , and that it carries a non-commutative product, which is, in a precise sense, “the most general product” on V .
The main idea of Seq2Tens is that any “static feature map” for elements in X
[Equation]
can be used to construct a new feature map Φ : Seq( X ) → T( V ) for sequences in X by using the algebraic structure of T( V ) : the non-commutative product on T( V ) makes it possible to “stitch together” the individual features φ ( x 1 ) , . .
, φ ( x L ) ∈ V ⊂ T( V ) of the sequence x in the larger space
T( V ) by multiplication in T( V ) .
With this we may deﬁne the feature map Φ( x ) for a sequences x = ( x 1 , . . .
, x L )
∈ Seq( X ) as follows
(i) lift the map φ : X → V to a map ϕ : X → T( V ) , (ii) map Seq( X ) → Seq(T( V )) by ( x 1 , . . .
, x L ) (cid:55)→ ( ϕ ( x 1 ) , . . .
, ϕ ( x L )) , (iii) map Seq(T( V ))
→ T( V ) by multiplication ( ϕ ( x 1 ) , . . . , ϕ
( x L ))
(cid:55)→ ϕ ( x 1 ) · · · ϕ ( x L ) .
In a more concise form, we deﬁne Φ as
[Equation]
where (cid:81) denotes multiplication in T( V ) .
We refer to the resulting map Φ as the Seq2Tens map, which stands short for Seq uences-
2 - Tens ors .
Why is this construction a good idea?
First note, that step (i) is always possible since V ⊂ T( V ) and we discuss the simplest such lift before Theorem 2.
as well as
other choices in Appendix B. Further, if φ , respectively ϕ , provides a faithful representation of objects in X , then there is no loss of information in step (ii).
Finally, since step (iii) uses “the most general product” to multiply ϕ ( x 1 ) · · · ϕ ( x L ) one expects that Φ( x ) ∈ T( V ) faithfully represents the sequence x as an element of T( V ) .
Indeed in Theorem 2.1 below we show an even stronger statement, namely that if the static feature map φ : X → V contains enough non-linearities so that non-linear functions from X to R can be approximated as linear functions of the static feature map φ , then the above construction extends this property to functions of sequences.
Put differently, if φ is a universal feature map for X , then Φ is a universal feature map for Seq( X ) ; that is, any non-linear function f ( x ) of a sequence x can be approximated as a linear functional of Φ( x ) , f ( x ) ≈ (cid:104) (cid:96), Φ( x ) (cid:105) .
We also emphasize that the domain of Φ is the space Seq( X ) of sequences of arbitrary (ﬁnite) length.
The remainder of this Section gives more details about steps (i),(ii),(iii) for the construction of Φ .
The free algebra T( V ) over a vector space V .
Let V be a vector space.
We denote by T( V ) the set of sequences of tensors indexed by their degree m ,
[Equation]
where by convention V ⊗
= R .
For example, if V = R d and t = ( t m ) m ≥ 0 is some element of
T( R d ) , then its degree m = 1 component is a d -dimensional vector t 1 , its degree m = 2 component is a d × d matrix t 2 , and its degree m = 3 component is a degree 3 tensor t 3 .
By deﬁning addition and scalar multiplication as
[Equation]
the set T( V ) becomes a linear space.
By identifying v ∈ V as the element (0 , v, 0 , . . . , 0) ∈ T( V ) we see that V is a linear subspace of T( V ) .
Moreover, while V is only a linear space, T( V ) carries a product that turns T( V ) into an algebra.
This product is the so-called tensor convolution product , and is deﬁned for s , t ∈ T( V ) as
[Equation]
where ⊗ denotes the usual outer tensor product; e.g. for vectors
u
= ( u i ) , v = ( v i ) ∈
R d the outer tensor product u ⊗ v is the d × d matrix ( u i v j ) i,j =1 ,...,d .
We emphasize that like the outer tensor product ⊗ , the tensor convolution product · is non-commutative, i.e. s · t (cid:54) =
t · s .
In a mathematically precise sense, T( V ) is the most general algebra that contains V ; it is a “free construction” .
Since
T( V ) is realized as series of tensors of increasing degree, the free algebra T( V ) is also known as the tensor algebra in the literature.
Appendix A contains background on tensors and further examples.
Lifting static feature maps.
Step (i) in the construction of Φ requires turning a given feature map φ : X → V into a map ϕ : X → T( V ) .
Throughout the rest of this article we use the lift
[Equation]
We discuss other choices in Appendix B, but attractive properties of the lift 6 are that (a) the evaluation of Φ against low rank tensors becomes a simple recursive formula (Proposition 3.3, (b) it is a generalization of sequence sub-pattern matching as used in string kernels (Appendix B.3, (c) despite its simplicity it performs exceedingly well in practice (Section 4).
Extending to sequences of arbitrary length.
Steps (i) and (ii) in the construction specify how the map Φ : X → T( V ) behaves on sequences of length- 1 , that is, single observations.
Step (iii) amounts to the requirement that for any two sequences x = ( x 1 , . . .
, x K ) ,
y = ( y 1 , . . .
, y L ) ∈
Seq( V ) , their concatenation deﬁned as z = ( x 1 , . . .
, x K , y 1 , . . .
, y L )
∈
Seq( V ) can be understood in the feature space as (non-commutative) multiplication of their corresponding features
[Equation]
In other words, we inductively extend the lift ϕ to sequences of arbitrary length by starting from sequences consisting of a single observation, which is given in equation 2.
Repeatedly applying the deﬁnition of the tensor convolution product in equation 5 leads to the following explicit formula
[Equation]
where x = ( x 1 , . .
, x L )
∈
Seq( V ) and the summation is over non-contiguous subsequences of x .
Some intuition: generalized pattern matching.
Our derivation of the feature map Φ( x ) =
(1 , Φ 1 ( x ) , Φ 2 ( x ) , . . . )
∈ T( V ) was guided by general algebraic principles, but equation 8 provides an intuitive interpretation.
It shows that for each m ≥ 1 , the entry Φ m ( x ) ∈ V ⊗ m constructs a summary of a long sequence x = ( x 1 , . . .
, x L )
∈
Seq( V ) based on subsequences ( x i 1 , . .
, x i m ) of x of length- m .
It does this by taking the usual outer tensor product x i 1 ⊗ · · · ⊗ x i m ∈ V ⊗ m and summing over all possible subsequences.
This is completely analogous to how string kernels provide a structured description of text by looking at non-contiguous substrings of length- m (indeed, Appendix B.3 makes this rigorous).
However, the main difference is that the above construction works for arbitrary sequences and not just sequences of discrete letters.
Readers with less mathematical background might simply take this as motivation and regard equation 8 as deﬁnition.
However, the algebraic background allows to prove that Φ is universal, see Theorem 2.1 below.
Universality.
A function φ : X → V is said to be universal for X if all continuous functions on
X can be approximated as linear functions on the image of φ .
One of the most powerful features of neural nets is their universality (Hornik, 1991).
A very attractive property of Φ is that it preserves universality: if φ : X → V is universal for X , then Φ : Seq( X ) → T( V ) is universal for Seq( X ) .
To make this precise, note that V ⊗ m is a linear space and therefore any (cid:96) =
( (cid:96) 0 , (cid:96) 1 , . . .
, (cid:96) M , 0 , 0 , . . . ) ∈
T( V ) consisting of M tensors (cid:96) m ∈ V ⊗ m , yields a linear functional on T( V ) ;
e.g. if V = R d and we identify (cid:96) m in coordinates as (cid:96) m =
( (cid:96) i 1 ,...,i m m )
i 1 ,...,i m ∈{ 1 ,...,d } then (cid:104) (cid:96), t (cid:105) :
=
M (cid:88) m = (cid:104) (cid:96) m , t m (cid:105) =
M (cid:88) m
= (cid:88)
i 1 ,...,i m ∈{ 1 ,...,d } (cid:96) i 1 ,...,i m m t i 1 ,...,i m m .
Thus linear functionals of the feature map Φ , are real-valued functions of sequences.
Theorem 2.
below shows that any continuous function f : Seq( X ) → R can by arbitrary well approximated by a (cid:96) ∈ T( V ) , f ( x ) ≈ (cid:104) (cid:96), Φ( x ) (cid:105) .
Theorem 2.1.
Let φ : X → V be a universal map with a lift that satisﬁes some mild constraints, then the following map is universal:
[Equation]
A detailed proof and the precise statement of Theorem 2.1 is given in Appendix B.
A PPROXIMATION BY LOW - RANK LINEAR FUNCTIONALS
The combinatorial explosion of tensor coordinates and what to do about it.
The universality of Φ suggests the following approach to represent a function f : Seq( X )
→ R of sequences:
First compute Φ( x ) and then optimize over (cid:96) (and possibly also the hyperparameters of φ ) such that f ( x ) ≈ (cid:104) (cid:96), Φ( x ) (cid:105) =
(cid:80)
Mm =
(cid:104) (cid:96) m ,
Φ m ( x ) (cid:105) .
Unfortunately, tensors suffer from a combinatorial explosion in complexity in the sense that even just storing Φ m ( x ) ∈ V ⊗ m ⊂ T( V ) requires O (dim( V ) m ) real numbers.
Below we resolve this computational bottleneck as follows: in Proposition 3.3 we show that for a special class of low-rank elements (cid:96) ∈ T( V ) , the functional x (cid:55)→ (cid:104) (cid:96), Φ( x ) (cid:105) can be efﬁciently computed in both time and memory.
This is somewhat analogous to a kernel trick since it shows that (cid:104) (cid:96), Φ( x ) (cid:105) can be cheaply computed without explicitly computing the feature map Φ( x ) .
However, Theorem 2.1 guarantees universality under no restriction on (cid:96) , thus restriction to rankfunctionals limits the class of functions f ( x ) that can be approximated.
Nevertheless, by iterating these “low-rank functional” constructions in the form of sequence-to-sequence transformations this can be ameliorated.
We give the details below but to gain intuition, we invite the reader to think of this iteration analogous to stacking layers in a neural network: each layer is a relatively simple non-linearity (e.g. a sigmoid composed with an afﬁne function) but by composing such layers, complicated functions can be efﬁciently approximated.
Rank-1 functionals are computationally cheap.
Degree m =
2 tensors are matrices and lowrank (LR) approximations of matrices are widely used in practice (Udell & Townsend, 2019) to address the quadratic complexity.
The deﬁnition below generalizes the rank of matrices (tensors of degree m = 2 ) to tensors of any degree m .
Deﬁnition 3.1.
The rank (also called CP rank (Carroll & Chang, 1970)) of a degree- m tensor t m ∈ V ⊗ m is the smallest number r ≥ 0 such that one may write
[Equation]
We say that t =
( t m ) m ≥ 0 ∈ T( V ) has rank-1 (and degree- M ) if each t m ∈ V ⊗ m is a rank-1 tensor and t i = [-
-]0 for i > M .
Remark 3.2.
For x = ( x 1 , . . .
, x L )
∈
Seq( V ) , the rank r m ∈ N of Φ m ( x ) satisﬁes r m ≤ (cid:0) Lm (cid:1) , while the rank and degree r, d ∈ N of Φ( x ) satisfy r ≤ (cid:0) LK (cid:1) for K = (cid:4) L 2 (cid:5) and d ≤ L .
A direct calculation shows that if (cid:96) is of rank- 1 , then (cid:104) (cid:96), Φ( x ) (cid:105) can be computed very efﬁciently by inner product evaluations in V .
Proposition 3.3.
Let (cid:96) =
( (cid:96) m )
m
≥ 0 ∈ T( V ) be of rank-1 and degree- M .
If φ is lifted to ϕ as in equation 6, then (cid:104) (cid:96), Φ( x ) (cid:105) =
[Equation]
(cid:88)
[Equation]
(12) where (cid:96) m
= v m 1 ⊗ · · · ⊗ v mm ∈ V ⊗ m , v m
i ∈ V and m = 0 , . .
, M .
Note that the inner sum is taken over all non-contiguous subsequences of x of length- m , analogously to m -mers of strings and we make this connection precise in Appendix B.3; the proof of Proposition 3.3 is given in Appendix B.1.1.
While equation 12 looks expensive, by casting it into a recursive formulation over time, it can be computed in O ( M 2 · L · d ) time and O ( M 2 · ( L + c )) memory, where d is the inner product evaluation time on V , while c is the memory footprint of a v ∈ V .
This can further be reduced to O ( M · L · d ) time and O ( M · ( L + c )) memory by an efﬁcient parametrization of the rank- 1 element (cid:96) ∈ T( V ) .
We give further details in Appendices D.2, D.3, D.4.
Low-rank Seq2Tens maps.
The composition of a linear map L : T( V ) → R N with Φ can be computed cheaply in parallel using equation 12 when L is speciﬁed through a collection of N ∈ N rank-1 elements (cid:96) 1 , . . .
, (cid:96) N ∈ T( V ) such that
[Equation]
We call the resulting map ˜Φ ˜ θ : Seq( X ) → R N a L ow-rank S eq 2T ens map of width- N and order- M , where M ∈ N is the maximal degree of (cid:96) 1 , . . .
, (cid:96) N such that (cid:96) ji = 0 for i > M .
The LS2T map is parametrized by (1) the component vectors v kj,m ∈ V of the rank- 1 elements (cid:96) jm = v 1 j,m ⊗· · ·⊗ v mj,m , (2) by any parameters θ that the static feature map φ θ : X → V may depend on.
We jointly denote these parameters by ˜ θ
= ( θ, (cid:96) 1 , . . .
, (cid:96) N ) .
In addition, by the subsequent composition of ˜Φ ˜ θ with a linear functional R N → R , we get the following function subspace as hypothesis class for the LS2T
[Equation]
Hence, we acquire an intuitive explanation of the (hyper)parameters: the width of the LS2T, N ∈ N speciﬁes the maximal rank of the low-rank linear functionals of Φ that the LS2T can represent, while the span of the rank- 1 elements, span( (cid:96) 1 , . . .
, (cid:96) N ) determine an N -dimensional subspace of the dual space of T( V ) consisting of at most rank- N functionals.
Recall now that without rank restrictions on the linear functionals of Seq2Tens features, Theorem 2.1 would guarantee that any real-valued function f : Seq( X ) → R could be approximated by f ( x ) ≈ (cid:104) (cid:96), Φ( x 1 , . . .
, x L ) (cid:105) .
As pointed out before, the restriction of the hypothesis class to lowrank linear functionals of Φ( x 1 , . .
, x L ) would limit the class of functions of sequences that can be approximated.
To ameliorate this, we use LS2T transforms in a sequence-to-sequence fashion that allows us to stack such low-rank functionals, signiﬁcantly recovering expressiveness.
Sequence-to-sequence transforms.
We can use LS2T to build sequence-to-sequence transformations in the following way: ﬁx the static map φ θ : X → V parametrized by θ and rank-
1 elements such that ˜ θ
= ( θ, (cid:96) 1 , . . .
, (cid:96) N ) and apply the resulting LS2T map ˜Φ ˜ θ over expanding windows of x :
[Equation]
Note that the cost of computing the expanding window sequence-to-sequence transform in equation 15 is no more expensive than computing ˜Φ ˜ θ ( x 1 , . . .
, x L ) itself due to the recursive nature of our algorithms, for further details see Appendices D.2, D.3, D.4.
Deep sequence-to-sequence transforms.
Inspired by the empirical successes of deep RNNs (Graves et al., 2013b;a; Sutskever et al., 2014), we iterate the transformation 15 D -times:
[Equation]
Each of these mappings Seq( R N i )
→ Seq( R N i +1 ) is parametrized by the parameters ˜
θ i of a static feature map φ θ i and a linear map L i speciﬁed by N i rank-1 elements of T( V ) ; these parameters are collectively denoted by ˜ θ i
=
( θ i , (cid:96) 1 i , . . .
, (cid:96) N i i ) .
Evaluating the ﬁnal sequence in Seq( R N D ) at the last observation-time t
=
L , we get the deep LS2T map with depth- D
[Equation]
Making precise how the stacking of such low-rank sequence-to-sequence transformations approximates general functions requires more tools from algebra, and we provide a rigorous quantitative statement in Appendix C. Here, we just appeal to the analogy made with adding depth in neural networks mentioned earlier and empirically validate this in our experiments in Section 4.
B UILDING NEURAL NETWORKS WITH LS2T LAYERS
The Seq2Tens map Φ built from a static feature map φ is universal if φ is universal, Theorem 2.1.
NNs form a ﬂexible class of universal feature maps with strong empirical success for data in X = R d , and thus make a natural choice for φ .
Combined with standard deep learning constructions, the framework of Sections 2 and 3 can build modular and expressive layers for sequence learning.
Neural LS2T layers.
The simplest choice among many is to use as static feature map φ :
X = R d → R h a feedforward network with depth- P , φ = φ P ◦· · ·◦ φ 1 where φ j ( x ) = σ ( W j x + b j ) for W j ∈ R h × d , b j ∈ R h .
We can then lift this to a map ϕ : R d → T( R h ) as prescribed in equation 6.
Hence, the resulting LS2T layer x (cid:55)→ (˜Φ ˜ θ ( x 1 , . . .
, x i ))
i =1 ,...,L is a sequence-to-sequence transform Seq( R d ) →
Seq( R h ) that is parametrized by ˜ θ =
( W 1 , b 1 , . . .
, W P , b P , (cid:96) 11 , . . .
, (cid:96) N 1 1 ) .
Bidirectional LS2T layers.
The transformation in equation 15 is completely causal in the sense that each step of the output sequence depends only on past information.
For generative models, it can behove us to make the output depend on both past and future information, see Graves et al.
(2013a);
Baldi et al.
; Li & Mandt (2018).
Similarly to bidirectional RNNs and LSTMs (Schuster &
Paliwal, 1997; Graves & Schmidhuber, 2005), we may achieve this by deﬁning a bidirectional layer, ˜Φ b(˜ θ 1 , ˜ θ 2 ) ( x ) :
Seq( R d ) →
Seq( R N + N (cid:48) ) , x (cid:55)→
[Equation]
θ 1 ( x 1 , . . . , x i ) , ˜Φ ˜ θ 2 ( x i , . . . , x L )) Li
=1 . The sequential nature is kept intact by making the distinction between what classiﬁes as past (the ﬁrst N coordinates) and future (the last N (cid:48) coordinates) information.
This amounts to having a form of precognition in the model, and has been applied in e.g. dynamics generation (Li & Mandt, 2018), machine translation (Sundermeyer et al., 2014), and speech processing (Graves et al., 2013a).
Convolutions and LS2T. We motivate to replace the time-distributed feedforward layers proposed in the paragraph above by temporal convolutions (CNN) instead.
Although theory only requires the preprocessing layer of the LS2T to be a static feature map, we ﬁnd that it is beneﬁcial to capture some of the sequential information in the preprocessing layer as well, e.g. using CNNs or RNNs.
From a mathematical point of view, CNNs are a straightforward extension since they can be interpreted as time-distributed feedforward layers applied to the input sequence augmented with a p ∈ N number of its lags for CNN kernel size p (see Appendix D.1 for further discussion).
In the following, we precede our deep LS2T blocks by one or more CNN layers.
Intuitively, CNNs and LS2Ts are similar in that both transformations operate on subsequences of their input sequence.
The main difference between the two lies in that CNNs operate on contiguous subsequences , and therefore, capture local, short-range nonlinear interactions between timesteps; while LS2Ts (equation 12) use all non-contiguous subsequences , and hence, learn global, long-range interactions in time.
This observation motivates that the inductive biases of the two types of layers (local/global time-interactions) are highly complementary in nature, and we suggest that the improvement in the experiments on the models containing vanilla CNN blocks are due to this complementarity.
E XPERIMENTS
We demonstrate the modularity and ﬂexibility of the above LS2T and its variants by applying it to (i) multivariate time series classiﬁcation, (ii) mortality prediction in healthcare, (iii) generative modelling of sequential data.
In all cases, we take a strong baseline model (FCN and GP-VAE, as detailed below) and upgrade it with LS2T layers.
As Thm.
2.1 requires the Seq2Tens layers to be preceded by at least a static feature map, we expect these layers to perform best as an add-on on top of other models, which however can be quite simple, such as a CNN.
The additional computation time is negligible (in fact, for FCN it allows to reduce the number of parameters signiﬁcantly, while retaining performance), but it can yield substantial improvements.
This is remarkable, since the original models are already state-of-the-art on well-established (frequentist and Bayesian) benchmarks.
5. M ULTIVARIATE TIME SERIES CLASSIFICATION
As the ﬁrst task, we consider multivariate time series classiﬁcation (TSC) on an archive of benchmark datasets collected by Baydogan (2015).
Numerous previous publications report results on this [Table] archive, which makes it possible to compare against several well-performing competitor methods from the TSC community. These baselines are detailed in Appendix E.1.
This archive was also considered in a recent popular survey paper on DL for TSC (Ismail Fawaz et al., 2019), from where we borrow the two best performing models as DL baselines: FCN and ResNet.
The FCN is a fully convolutional network which stacks 3 convolutional layers of kernel sizes (8 , 5 , 3) and ﬁlters (128 , 256 , 128) followed by a global average pooling (GAP) layer, hence employing global parameter sharing. We refer to this model as FCN 128 . The ResNet is a residual network stacking 3 FCN blocks of various widths with skip-connections in between (He et al., 2016) and a ﬁnal GAP layer. The FCN is an interesting model to upgrade with LS2T layers, since the LS2T also employs parameter sharing across the sequence length, and as noted previously, convolutions are only able to learn local interactions in time, that in particular makes them ill-suited to picking up on longrange autocorrelations, which is exactly where the LS2T can provide improvements.
As our models, we consider three simple architectures: (i) LS2T [Equation]
stacks 3 LS2T layers of order- 2 and width- 64 ; (ii) FCN 64 -LS2T 364 precedes the LS2T 364 block by an FCN 64 block; a downsized version of FCN 128 ; (iii) FCN 128 -LS2T 364 uses the full FCN 128 and follows it by a LS2T 364 block as before. Also, both FCN -LS2T models employ skip-connections from the input to the LS2T block and from the FCN to the classiﬁcation layer, allowing for the LS2T to directly see the input
and for the FCN to directly affect the ﬁnal prediction.
These hyperparameters were only subject to hand-tuning on a subset of the datasets, and the values we considered were H, N ∈ { 32 , 64 , 128 } , M ∈ { 2 , 3 , 4 } and D ∈ { 1 , 2 , 3 } , where H, N ∈ N is the FCN and LS2T width, resp., while M ∈ N is the LS2T order and D ∈ N is the LS2T depth.
We also employ techniques such as time-embeddings (Liu et al., 2018a), sequence differencing and batch normalization, see Appendix D.1; Appendix E.1 for further details on the experiment and Figure 2 in thereof for a visualization of the architectures.
Results.
We trained the models, FCN 128 , ResNet, LS2T 364 , FCN 64 -LS2T 364 , FCN 128 -LS2T 364 on each of the 16 datasets 5 times while results for other methods were borrowed from the cited publications.
In Appendix E.1, Figure 3 depicts the box-plot of distributions of accuracies and a CD diagram using the Nemenyi test (Nemenyi, 1963), while Table 7 shows the full list of means and standard deviations.
Since mean-ranks based tests raise some paradoxical issues (Benavoli et al., 2016), it is customary to conduct pairwise comparisons using frequentist (Demˇsar, 2006) or Bayesian (Benavoli et al., 2017) hypothesis tests.
We adopted the Bayesian signed-rank test from Benavoli et al.
(2014), the posterior probabilities of which are displayed in Table 1, while the Bayesian posteriors are visualized on Figure 4 in App.
E.1.
The results of the signed-rank test can be summarized as follows: (1) LS2T 364 already outperforms some classic TS classiﬁers with high probability ( p ≥ 0 . 8 ), but it is not competitive with other DL classiﬁers. This observation is not surprising since even theory requires at least a static feature map to precede the LS2T. (2) FCN 64 -LS2T 364 outperforms almost all models with high probability ( p ≥ 0 . 8 ), except for ResNet (which is stil outperformed by p ≥ 0 . FCN
and FCN -LS2T When
compared with FCN 128 , the test is unable to decide between the two, which
upon inspection of the {+individual results in Table 
+} can be explained by that on some datasets the beneﬁt of the added LS2T block is high enough that it outweighs the loss of ﬂexibility incurred by reducing the width of the FCN - arguably these are the datasets where long-range auto-
correlations are present in the input time series, and picking up on these improve the performancehowever, on a few datasets the opposite is true. (3) Lastly, FCN 128 -LS2T 364 , outperforms all baseline methods with high probability ( p ≥ 0 . 8 ) , and hence successfully improves on the FCN 128 via its added ability to learn long-range time-interactions.
We remark that FCN 64 -LS2T 364 has fewer parameters than FCN by more than 60% , hence we managed to compress it to a fraction of its original size, while on average, still improving on its performance, a nontrivial feat by its own accord.
5. M ORTALITY PREDICTION
We consider the P HYSIONET 2012 challenge dataset (Goldberger et al., 2000) for mortality prediction, which is a case of medical TSC as the task is to predict in-hospital mortality of patients after their admission to the ICU.
This is a difﬁcult ML task due to missingness in the data, low signalto-noise ratio (SNR), and unbalanced class distributions with a prevalence ratio of around 14% .
We extend the experiments conducted in Horn et al.
(2020), which we also use as very strong baselines. Under the same experimental setting, we train two models: FCN-LS2T as ours and the FCN as another baseline.
For both models, we conduct a random search with 20 samples for all hyperparameters from a pre-speciﬁed search space, and the setting with best validation performance is used for model evaluation on the test set over 5 independent model trains, exactly the same way as it was done in Horn et al.
We preprocess the data using the same method as in Che et al.
(2018, eq.
and additionally handle static features by tiling them along the time axis and adding them as extra coordinates.
We additionally introduce in both models a SpatialDropout1D layer after all CNN and LS2T layers with the same tunable rate to mitigate the low SNR of the dataset. -LS2T and FCN on P HYSIONET 2012 with the results from Horn et al.
[Table]
Results.
Table 2 compares the performance of FCN-LS2T with that of FCN and the results from Horn et al.
(2020) on 3 metrics: (1) ACCURACY , (2) area under the precision-recall curve ( AUPRC ), (3) area under the ROC curve ( AUROC ).
We can observe that FCN-LS2T takes on average overall ﬁrst place according to both ACCURACY and AUPRC , outperforming FCN and all SOTA methods , e.g. T RANSFORMER (Vaswani et al., 2017), GRU-D Che et al. (2018), S E FT (Horn et al., 2020), and also being competitive in terms of
AUROC .
This is very promising, and it suggests that LS2T layers might be particularly well-suited to complex and heterogenous datasets, such as medical time series, since the FCN-LS2T models signiﬁcantly improved accuracy on ECG as well, another medical dataset in the previous task.
5. G ENERATING SEQUENTIAL DATA
Finally, we demonstrate on sequential data imputation for time series and video that LS2Ts do not only provide good representations of sequences in discriminative, but also generative models.
The GP-VAE model.
In this experiment, we take as base model the recent GP-VAE (Fortuin et al., 2020), that provides state-of-the-art results for probabilistic sequential data imputation.
The
GP-VAE is essentially based on the HI-VAE (Nazabal et al., 2018) for handling missing data in variational autoencoders (VAEs) (Kingma & Welling, 2013) adapted to the handling of time series data by the use of a Gaussian process (GP) prior (Williams & Rasmussen, 2006) across time in the latent sequence space to capture temporal dynamics.
Since the GP-VAE is a highly advanced model, its in-depth description is deferred to Appendix E.2.We extend the experiments conducted in
Fortuin et al.
and we make one simple change to the GP-VAE architecture without changing any other hyperparameters or aspects: we introduce a single bidirectional LS2T layer (B-LS2T) into the encoder network that is used in the amortized representation of the means and covariances of the variational posterior.
The B-LS2T layer is preceded by a time-embedding and differencing block, and succeeded by channel ﬂattening and layer normalization as depicted in Figure 5.
The idea behind this experiment is to see if we can improve the performance of a highly complicated model that is composed of many interacting submodels, by the naive introduction of LS2T layers.
[Table]
Results.
To make the comparison [Table] we ceteris paribus re-ran all experiments the authors originally included in their paper (Fortuin et al., 2020), which are imputation of Healing MNIST, Sprites, and Physionet 2012.
The results are in Table
[Table]
which report the same metrics as used in Fortuin et al.
(2020), i.e. negative log-likelihood (NLL, lower is better), mean squared error (MSE, lower is better) on test sets
[Table]
and downstream classiﬁcation performance of a linear classiﬁer (AUROC, higher is better).
For all other models beside our GP-VAE (B-LS2T), the results were borrowed from Fortuin et al.
We observe that simply adding the B-LS2T layer improved the result in almost all cases, except for Sprites, where the GP-VAE already achieved a very low MSE score.
Additionally, when comparing GP-VAE to BRITS on Physionet, the authors argue that although the
BRITS achieves a higher AUROC score, the GP-VAE should not be disregarded as it ﬁts a generative model to the data that enjoys the usual Bayesian beneﬁts of predicting distributions instead of point predictions.
The results display that by simply adding our layer into the architecture, we managed to elevate the performance of GP-VAE to the same level while retaining these same beneﬁts.
We believe the reason for the improvement is a tighter amortization gap in the variational approximation (Cremer et al., 2018) achieved by increasing the expressiveness of the encoder by the LS2T allowing it to pick up on long-range interactions in time.
We provide further discussion in Appendix E.2.
R ELATED WORK AND S UMMARY
Related Work.
The literature on tensor models in ML is vast.
Related to our approach we mention pars-pro-toto Tensor Networks (Cichocki et al., 2016), that use classical LR decompositions, such as CP (Carroll & Chang, 1970), Tucker (Tucker, 1966), tensor trains (Oseledets, 2011) and tensor rings (Zhao et al., 2019); further, CNNs have been combined with LR tensor techniques (Cohen et al., 2016; Kossaiﬁ et al., 2017) and extended to RNNs (Khrulkov et al., 2019); Tensor Fusion
Networks (Zadeh et al., 2017) and its LR variants (Liu et al., 2018b; Liang et al., 2019; Hou et al., 2019); tensor-based gait recognition (Tao et al., 2007).
Our main contribution to this literature is the use of the free algebra T( V ) with its convolution product · , instead of V ⊗ m with the outer product ⊗ that is used in the above papers.
While counter-intuitive to work in a larger space T( V ) , the additional algebra structure of (T( V ) , · ) is the main reason for the nice properties of Φ ( universality, making sequences of arbitrary length comparable, convergence in the continuous time limit ; see
Appendix B) which we believe are in turn the main reason for the strong benchmark performance .
Stacked LR sequence transforms allow to exploit this rich algebraic structure with little computational overhead.
Another related literature are path signatures in ML (Lyons, 2014;
Chevyrev &
Kormilitzin, 2016; Graham, 2013; Bonnier et al., 2019; Toth & Oberhauser, 2020).
These arise as special case of Seq2Tens (Appendix B) and our main contribution to this literature is that Seq2Tens resolves a well-known computational bottleneck in this literature since it never needs to compute and store a signature , instead it directly and efﬁciently learns the functional of the signature .
Summary.
We used a classical non-commutative structure to construct a feature map for sequences of arbitrary length.
By stacking sequence transforms we turned this into scalable and modular NN layers for sequence data.
The main novelty is the use of the free algebra T( V ) constructed from the static feature space V .
While free algebras are classical in mathematics, their use in ML seems novel and underexplored.
We would like to re-emphasize that (T( V ) , · ) is not a mysterious abstract space: if you know the outer tensor product ⊗ then you can easily switch to the tensor convolution product · by taking sums of outer tensor products, as deﬁned in equation 5.
As our experiments show, the beneﬁts of this algebraic structure are not just theoretical but can signiﬁcantly elevate performance of already strong-performing models.