Empirics on the Expressiveness of Randomized Signature
Abstract
Time series analysis is a widespread task in Natural Sciences, Social Sciences and Engineering.
A fundamental problem is ﬁnding an expressive yet efﬁcientto-compute representation of the input time series to use as a starting point to perform arbitrary downstream tasks.
In this paper, we build upon recent work using signature of a path as a feature map, and investigate a computationally efﬁcient technique to approximate these features based on linear random projections.
We present several theoretical results to justify our approach, we analyze and showcase its empirical performance on the task of learning a mapping between the input controls of a Stochastic Differential Equation (SDE) and its corresponding solution.
Our results show that the representational power of the proposed random features allows to efﬁciently learn the aforementioned mapping.
Introduction
Modeling time series is a common task in ﬁnance, physics, and engineering.
Differential (or
Difference) equations are the most widespread approach, as they allow to specify the controls of the dynamics (i e. external ﬁelds, time, input signals, noise, etc.) as well as their precise inﬂuence.
Unfortunately, guessing the true algebraic expression of such equations from ﬁrst principles considerations is often impossible.
Given a dataset of several input-output pairs, one can approach the problem from a data-driven perspective and construct a surrogate of the underlying operator mapping controls (inputs) to output.
However, the size of the available datasets is not often sufﬁciently large to support the requirements of modern over-parametrized deep learning algorithms [Brown et al., 2020,
Kaplan et al., 2020].
Additionally, these techniques do not come with guarantees on their ability to extract provably robust features from the input paths, and their training can be highly computationally demanding.
In this work we build on ideas from rough path theory and reservoir computing
[Friz and Hairer, 2020, Schrauwen et al., 2007] to extract a highly expressive set of (random) features from the input controls, allowing efﬁcient learning of the dynamics of complicated SDEs.
The features we consider are obtained by integrating over a ﬁxed time window a latent SDE whose dynamic is modulated by linear random vector ﬁelds and driven by the input controls.
The output, i.e. the solution to the original SDE, is simply given by a trained linear projection of the random features.
This approach presents several advantages: a) Our random features provably approximate the signature of the control paths.
In particular, we circumvent their potentially expensive calculation (see e.g. Lyons [2014], Kidger and Lyons [2020]) preserving their information content and geometrical properties (see also discussion in the next section); b) The features we extract are non-local, in the sense that they are the result of the evolution of an SDE over a ﬁxed time window and therefore encode information about the whole considered time interval; c) Contrarily to many deep neural networks for time series analysis, our approach is not limited to uniformly sampled sequences [Fawaz et al., 2019].
This paper is structured as follows:
In Section 2, we present some theoretical insights motivating our method, in Section 3, we empirically validate the performance of our method on various simulated datasets, in Section 4 with link the proposed approach with related works in the literature, and in
Section 5, we discuss our conclusion and possible future work.
Background 2.1 Randomized Signature of a Path
Let X : [0 , T ] → R d be a continuous piecewise smooth d − dimensional path X
= (cid:0) X 1 , · · · , X d (cid:1) .
We will refer to X as the control and to its single components
X i as controls .
We denote by { e 1 , . . .
, e d } the canonical basis of R d .
Deﬁnition 1 (Signature)
For any t ∈ [0 , T ] , the Signature of a continuous piecewise smooth path
X : [0 , T ] → R d on [0 , t ] is the countable collection S X [0 ,t ] := (cid:16) 1 , S X, 1 [0 ,t ] , S X, 2 [0 ,t ] , . . .
(cid:17) ∈ (cid:81) ∞
k =0 (cid:0) R d (cid:1) ⊗ k where, for each k ≥ 1 , the entries S X,k [0 ,t ] are the iterated integrals deﬁned as
S X,k [0 ,t ] := (cid:88) ( i
[Equation]
{+{ 1 ,...,d } k (cid:18)(cid:90) 
+}
≤ s 1 ≤···≤ s k ≤ t dX i 1 s 1 . . . dX i k s k (cid:19) e i 1 ⊗ · · · ⊗ e i k .
We deﬁne the Truncated Signature of X of order M ≥ 0 as S X,M [0 ,t ] := (cid:16) 1 , S X, 1 [0 ,t ] , . . .
, S X,M [0 ,t ] (cid:17) ∈ (cid:81) Mk =0 (cid:0) R d (cid:1) ⊗ k =: T M (cid:0) R d (cid:1) .
Remark
We highlight that S X,k [0 ,t ] lives in (cid:0) R d (cid:1) ⊗ k , which is the space of tensors of shape ( d, . . .
, d ) given by R d ⊗ · · · ⊗ R d for k times.
Theorem 1 (Signature is a Reservoir)
Let V i : R m → R m , i = 1 , . . .
, d be vector ﬁelds regular enough such that dY t = (cid:80) di
=1 V i
( Y t )
dX it , Y 0 =
y ∈ R m , admits a unique solution Y t : [0 , T ] →
R m .
Then, for any smooth test function F : R m → R and for every M ≥
there is a timehomogeneous linear operator L : T M (cid:0) R d (cid:1)
→ R which depends only on ( V 1 , . .
, V d , F, M, y ) such that F ( Y t ) =
L (cid:16) S X,M [0 ,t ] (cid:17) + O (cid:0) t M +1 (cid:1) , and t ∈ [0 , T ] .
This theorem suggests the ﬁrst M entries of the signature of X are sufﬁcient to linearly explain the solution of any differential equation driven by it.
Unfortunately, calculating S X,M [0 ,t ] requires the calculation of d M +1 − 1 d −iterated integrals – which in total quickly becomes computationally expensive.
Of course several techniques have been developed to circumvent this problem, in particular kernelization techniques, see, e.g., Kidger and Lyons [2020].
The next result provides a practical description of how it is possible to reduce the computational burden without losing too much explanatory power.
The formal statements are provided in Appendix.
Theorem 2 (Randomized Signature (Informal))
For any k ∈ N big enough and appropriately chosen random matrices
A 1 , . . . ,
A d in R k × k and random shifts b 1 , . . .
, b d in R k × 1 , and any ﬁxed sigmoid function σ , the solution of
[Equation]
approximation signature
To conclude, we highlight that the computational complexity of calculating Z Xt is k
2 × d for each time step.
Experiments 3.1 1 -Dimensional Stochastic Double Well
Let us recall that the dynamics of the 1 -Dimensional Stochastic Double Well process is given by
[Equation]
where W t is a 1 -dimensional Brownian motion, and ( µ, θ, σ ) ∈ R × R + × R + .
Let us ﬁx y
= 1 and ( µ = 2 , θ = 1 , σ = 1) , and the partition D of [0 , 1] to have N = 101 equally spaced time steps.
We train a Ridge Regression with regularization parameter λ = 0 .
001 to map instances of Randomized
Signature of the controls Z [ t,W t ] into the respective solution Y t .
We repeat the experiment on different values of the number N Train of train samples and dimension k of the Z [ t,W t ] .
On the left of Figure 1, we plot an example of the trajectory of Z [ t,W t ] while, on its right, we plot the comparison of the true and the generated time series on an out of sample case.
The following table shows the performance in terms of L 2 relative error on 10000 test samples:
[Equation] k
[Equation]
k =0.
[Table] [Table] [Table] [Table]
k =
[Table] [Table] [Figure]
Sample Extracted Y t True Y t
[Figure]
Let us recall that the dynamics of the 4 -Dimensional Ornstein–Uhlenbeck process is given by
[Equation]
where W t is a 4 -dimensional Brownian motion, and ( µ , Θ , Σ ) ∈ R 4 × R 4 × 4 × R 4 × 4 .
Let us {+ﬁx y 
+} = 1 , µ = 1 ,
Σ = 1 4 , Θ i.j = i/j , the partition D of [0 , 1] to have N = 101 equally spaced time steps, and k = 708 .
Finally, we train a Ridge Regression with λ = 0 .
001 on N Train train sample and
Figure 2 shows the comparison of Out of Sample generated and true trajectories while Table 3 in
Appendix reports the performance in terms of L 2 relative error on 10000 test samples.
3.3 1 -Dimensional Stochastic Double Well - Irregularly Sampled Time Grid
In this experiment, we ﬁx y
= 1 and ( µ = 2 , θ = 1 , σ = 1) .
On the other hand, for each train and test sample, the partition D of [0 , 1] is made of N randomly drawn times.
More precisely, D = { 0 , t 1 , · · · , t N − 1 , 1 } such that t k = 1 / (1 − exp( − s k )) and { s 1 , · · · , s N − 1 } are N − 2 independent realizations of a uniform distribution U [0 , 1] sorted increasingly.
As a result, the probability that two
0.0 0.2 0.4 0.6 0.8 1.0 Time 0.
V a l u e
Extracted Y 1 t
True Y 1 t 0.0 0.2 0.4 0.6 0.8 1.0 Time 0.
0.0 0.2 0.4 0.6 0.8 1.0 Time 0. 0.0 0.2 0.4 0.6 0.8 1.0 Time 0.
samples share the same D is null.
We train a Ridge Regression with λ = 0 .
001 on N Train =
train samples and Figure 3 in Appendix shows the comparison of an Out of Sample generated and true trajectory.
Finally, Table 2 shows the Relative L 2 Error on 10000 test samples as we vary the number of time steps N and k and compare it to the respective experiment in case the time grid is regularly spaced.
( N, k )
=
(10 , 56) ( N, k )
= (100 , 222) ( N, k ) =
[Table]
Related Works
Random Features and Reservoir Computing.
The idea of extracting features based on random operations is not new and has seen a number of successful applications over the past years.
On particular note for our method, the seminal work of Rahimi and Recht [2008] proposes to accelerate kernel machines by learning random features whose inner product matches that of a target shift-invariant kernel.
The trade-off between generalization and computational efﬁciency of learning with random features has then been rigorously studied by Rudi and Rosasco [2017].
A conceptually very similar rationale is introduced by a parallel series of works exploring the topic of Reservoir Computing [Schrauwen et al., 2007].
Similarly to our work, Echo State Networks [Jaeger, 2003] evolve the input state by a series of ﬁxed random projection (the reservoir) and generate the output by applying a trainable linear projection over the hidden states.
However, we make the additional step of linking the random features to the signature of the input path and, in our case, the evolution of the features is dictated by the differential equation shown in equation 1.
Controlled Differential Equations.
Our work is also related with a series of recent papers investigating the problems of how to process irregular time series and to condition a model on incoming information through the lens of controlled differential equations [Kidger et al., 2020, Morrill et al., 2021].
Differently from them, our method is way more efﬁcient since the only parameters we need to train are those of the ﬁnal (often) linear readout layer.
Rough Path Theory.
Rough path theory is about describing how possibly highly oscillatory (rough) control path interact with nonlinear systems [Lyons, 2014].
The concept of signature is introduced in this context to provide a powerful description of the input path upon which the theory is built.
Recent years have seen a resurgence of these ideas, which have been revisited from a machine learning perspective [Bonnier et al., 2019, Kidger and Lyons, 2020].
Our analysis is strongly inﬂuenced by the work of Cuchiero et al.
[2021b], who ﬁrstly establishes a connection between reservoir computing and signature of a paths (in a discrete setting).
Conclusions
In this work, we empirically analyze an approach based on random features to ﬁnd a powerful representation of an input path.
Our randomly extracted features approximate the signature of the given path and thus ﬁnd theoretical support in the theory of rough path.
We empirically assess the effectiveness of our method by inspecting its performance on several tasks based on learning a map between possibly multivariate and irregularly sampled input controls into the solution of the corresponding SDEs.
A Appendix A.1 Basic Deﬁnitions
First of all, we deﬁne the concept of admissible tensor norms, which we assume to have in this work.
Deﬁnition 2 (Admissible Tensor Norms)
Let E := R d and ⊗ be a tensor product such that the tensor powers of E , (cid:0) E ⊗ k : k ≥ 1 (cid:1) , are equipped with a family ( (cid:107) · (cid:107) E ⊗ k : k ≥ 1) of norms satisfying:
For j, k ∈ N and for all h ∈ E ⊗ j and l ∈ E ⊗ k (cid:107) h ⊗ l (cid:107) E ⊗ ( j + k ) ≤ (cid:107) h (cid:107) E ⊗ j (cid:107) l (cid:107) E ⊗ k ; 2.
For any permutation σ of { 1 , . . . , k } ,
[Equation]
[Equation] [Equation] For any bounded linear functionals f on E ⊗ j and g on E ⊗ k , there exists a unique bounded linear functional, denoted as f ⊗ g , on E ⊗ ( j + k ) such that for all h ∈ V ⊗ j and l ∈ E ⊗ k f ⊗ g ( h ⊗ l ) =
f ( h ) g
( l ) .
A family of tensor norms satisfying these conditions is called a family of admissible tensor norms.
Then we deﬁne the space in which the signature of a path lies in, that is the following Tensor Algebra.
Deﬁnition 3 (Tensor Algebra T (cid:0) R d (cid:1) )
We deﬁne the tensor algebra on R d as
[Equation]
as well as its truncated version of order M ≥ 0 as
[Equation]
where (cid:0) R d (cid:1) ⊗ k is the space of tensors of shape ( d, . . .
, d ) given by R d ⊗ · · · ⊗ R d for k times.
Deﬁnition 4 (Concatenation Operation)
We deﬁne the Concatenation Operation ∗ such that for any given couple of continuous piecewise smooth paths X : [0 , s ] → R d and Y : [ s, t ] → R d , their image through ∗ is the continuous piecewise smooth path X ∗ Y : [0 , t ] → R d deﬁned by
[Equation]
Deﬁnition 5 (Inverse Operation)
We deﬁne the Inverse Operation ←− such that for any continuous piecewise smooth path X : [0 , T ] → R d , its image through ←− is the continuous piecewise smooth path ←−
X t := X T − t , for each t ∈ [0 , T ] .
Deﬁnition 6 (Topological Space)
A topological space is an ordered pair ( X, τ ) , where X is a set and τ is a collection of subsets of X , satisfying the following axioms:
The empty set and X itself belong to τ ; 2.
Any arbitrary (ﬁnite or inﬁnite) union of members of τ still belongs to τ ; 3.
The intersection of any ﬁnite number of members of τ still belongs to τ .
The elements of τ are called open sets and the collection τ is called a topology on X .
Deﬁnition 7 (Arcwise Connected Topological Space)
A topological space ( X, τ ) is said to be arcwise connected if any two distinct points x, y ∈ X can be joined by an arc, that is a continuous map α : [0 , 1] → X such that α (0) =
x and α (1) =
y .
We say that X is uniquely arcwise connected if for any two distinct points x,
y ∈ X , there exists a unique path in X that joins them.
Deﬁnition 8 ( R -tree)
An R -tree is a uniquely arcwise connected metric space, in which the arc between two points is isometric to an interval.
Deﬁnition 9 (Tree-Like)
A continuous piecewise smooth path x : [0 , T ] → R d is tree-like if there exists an R -tree τ , a continuous map φ : [0 , T ] → τ and a map ψ : τ → V such that φ (0) =
φ ( T ) and x = ψ ◦ φ .
As T. Levy showed in Section 5.7 of [], a continuous piecewise smooth path is tree-like if and only if it is contractible to the constant path within its own image.
A.2 Supporting Results
Theorem 3 Given a continuous piecewise smooth path X : [0 , T ] → R d , its Signature is the banal signature 1 := (1 , 0 , 0 , · · · ) if and only if X is tree-like.
In particular S ( X ∗ ←− X ) = 1 .
Lemma 4 (Johnson-Lindenstrauss Lemma)
Given an M -dimensional Hilbert space ( H, (cid:104)· , ·(cid:105) H ) and Q
a N -point subset of H , for any 0 < (cid:15) < 1 , for each k ∈ N satisfying the so called JohnsonLindenstrauss constraint
[Equation]
there exists a linear map f { Q,M,k } : H → R k that embeds Q into R k in an almost isometric manner.
More speciﬁcally, we have that
[Equation]
[Table]
for each a 1 , a 2 ∈ Q .
Remark 2 When there is no need to specify the dependence of f { Q,M,k } from Q , M , or k , we will omit them up to referring to is simply as f .
Finally, we deﬁne f ∗ :
R k
→ H to be the adjoint map of f with respect to a ﬁxed inner product (cid:104)· , ·(cid:105) in R k .
To apply such Lemma in our context, we select M ≥ 0 , equip T M (cid:0) R d (cid:1) with an inner product such that
[Equation]
where { e i 1 ⊗ · · · ⊗ e i M } i 1 ,...,i M ∈{ 1 ,...,d } is the canonical basis of T M (cid:0) R d (cid:1) .
Therefore, we have that (cid:0)
T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) is an Hilbert space.
Theorem 5 (Existence and Uniqueness of the Signature)
The following controlled differential equation
[Equation]
has a unique solution S Xt which, at each t ∈ [0 , T ] is the signature S X [0 ,t ] of X on [0 , t ] .
Notation
In the light of the previous result, if there is no ambiguity about [0 , t ] , we will often refer to S X [0 ,t ] as S X and as S Xt to stress its path-like nature.
Analogously, we will use S X,M and S X,Mt in place of S X,M [0 ,t ] .
Now, we provide results to show the relevance of these features.
First of all, the following theorem ensures that S X encodes the essence of X and characterizes it completely.
According to Theorem 3, the concatenation X ∗←− X of X with its inverse 1 ←− X has the same signature as the constant path, but cannot be reparametrised to be constant.
Similarly, if X, Y, Z are non-constant paths, then S X ∗ Y ∗←− Y ∗ Z ∗←− Z ∗←− X = 1 , but X ∗ Y ∗
←− Y ∗ Z ∗ ←− Z ∗ ←− X is not a path of the form γ ∗ ←− γ for any path γ .
While the formal deﬁnition of tree-like path is given in Deﬁnition 10, Figure 1 guides our intuition as we notice that these paths look like trees and can be reduced to a constant path by removing possibly inﬁnitesimal pieces of the form γ ∗ ←− γ .
Theorem 6 (Characterizing Nature of the Signature)
Given a couple of continuous piecewise smooth paths X and ˆ X , then S X = S
ˆ X
if and only if X ∗ ←− ˆ X is tree-like.
This result is actually much stronger as it implies that the solution of any differential equation controlled by X is fully determined by the vector ﬁelds and S X .
In particular, Theorem 1 shows that the solution of any differential equation controlled by X is essentially linear in S X .
Adapting Theorem III.7 in [Cuchiero et al., 2021b]
it can be be shown that (asymptotically) the JL projected vector ﬁelds stem from random matrices:
Theorem 7 ( Z X is a random dynamical system)
For k (cid:55)−→ ∞ , for each i ∈ { 1 , · · · , d } the linear vector ﬁelds f { k } (cid:0) f { k }∗ ( · ) e i (cid:1) : R k
→ R k are square matrices with asymptotically normally distributed, independent entries.
Adapting Theorem III.8.
[Cuchiero et al., 2021b] as done in [Cuchiero et al., 2021a] one obtains:
Theorem 8 (Randomized Signature)
For any ﬁxed integer M ≥ 0 , any ﬁxed partition D = { t 1 , · · · , t N } of [0 , T ] such that 0 ≤ t 1 < · · · < t N ≤ T , let us consider the Truncated Signature of
X at such times, that is Q :=
(cid:110) S X,Mt, · · · , S X,Mt N (cid:111) such that its elements all lie in (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) .
Let us now select 0 < (cid:15) < 1 , k ∈ N satisfying the associated Johnson-Lindenstrauss constraint, let f be the implied Johnson-Lindenstrauss map and f ∗ its adjoint map.
Then, the solution of the controlled differential equation in R k
[Equation]
on D , that is (cid:8) Z Xt 1 , · · · , Z Xt N (cid:9) , are called the Randomized Signature of X on D .
Each Z Xt k is the projection of S X,Mt k from (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) onto R k via f and thus preserves its geometric properties and its approximation power.
Deﬁnition 10 (Localized Randomized Signature)
For any random matrices A 1 , . . .
, A d in R k × k and shifts b 1 , . . .
, b d in R k × 1 such that maximal non-integrability holds on a starting point z ∈ R k , any ﬁxed sigmoid function σ , and d -dimensional control X , the solution of
[Equation]
is called the localized randomized signature of X .
[Equation]
A.3 Experiments: Supporting material
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table] [Figure]