Original: R ANDOMIZED S IGNATURE L AYERS FOR S IGNAL E XTRACTION IN T IME S ERIES D ATA
Revised: R ANDOMIZED S IGNATURE L AYERS FOR S IGNAL E XTRACTION IN T IME S ERIES D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Time series analysis is a widespread task in Natural Sciences, Social Sciences, and Engineering.
Revised: Time series analysis is a widespread task in Natural Sciences, Social Sciences, and Engineering.

Original: A fundamental problem is ﬁnding an expressive yet efﬁcient-tocompute representation of the input time series to use as a starting point to perform arbitrary downstream tasks.
Revised: A fundamental problem is ﬁnding an expressive yet efﬁcient-tocompute representation of the input time series to use as a starting point to perform arbitrary downstream tasks.

Original: In this paper, we build upon recent works that use the
Revised: In this paper, we build upon recent works that use the

Original: Signature of a path as a feature map and investigate a computationally efﬁcient technique to approximate these features based on linear random projections.
Revised: Signature of a path as a feature map and investigate a computationally efﬁcient technique to approximate these features based on linear random projections.

Original: We present several theoretical results to justify our approach and empirically validate that our random projections can effectively retrieve the underlying Signature of a path.
Revised: We present several theoretical results to justify our approach and empirically validate that our random projections can effectively retrieve the underlying Signature of a path

Original: We show the surprising performance of the proposed random features on several tasks, including (1) mapping the controls of stochastic differential equations to the corresponding solutions and (2) using the Randomized Signatures as time series representation for classiﬁcation tasks.
Revised: We show the surprising performance of the proposed random features on several tasks including (1) mapping the controls of stochastic differential equations to the corresponding solutions and (2) using the Randomized Signatures as time series representation for classiﬁcation tasks.

Revised: When compared to corresponding truncated Signature approaches, our Randomizes Signatures are more computationally efﬁcient in high dimensions and often lead to better accuracy and faster training.

Original: Besides providing a new tool to extract Signatures and further validating the high level of expressiveness of such features, we believe our results provide interesting conceptual links between several existing research areas, suggesting new intriguing directions for future investigations.
Revised: Besides providing a new tool to extract Signatures and further validating the high level of expressiveness of such features, we believe our results provide interesting conceptual links between several existing research areas, suggesting new intriguing directions for future investigations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modeling time series is a common task in ﬁnance, physics, and engineering.
Revised: Modeling time series is a common task in ﬁnance, physics, and engineering.

Original: A frequent challenge is ﬁnding a transformation mapping a newly observed time series into a target one (seq2seq modeling) or into a label summarizing its salient properties (classiﬁcation).
Revised: A frequent challenge is ﬁnding a transformation mapping a newly observed time series into a target one (seq2seq modeling) or into a label summarizing its salient properties (classiﬁcation).

Original: In the absence of any principled model describing such a mapping, one has to infer it from data.
Revised: In the absence of any principled model describing such a mapping, one has to infer it from data.

Original: The last few years have witnessed the rise of deep neural networks, which have found successful application to problems involving time series in numerous domains (Fawaz et al., 2019; Gamboa, 2017).
Revised: The last few years have witnessed the rise of deep neural networks, which have found successful application to problems involving time series in numerous domains (Fawaz et al., 2019; Gamboa, 2017).

Original: Nevertheless, their outstanding performance comes at the price of over-parametrization, data hungriness, and expensive training cost (Werbos, 1990; Brown et al., 2020; Teubert et al., 2021; Neyshabur et al., 2018; Marcus, 2018).
Revised: Nevertheless, their outstanding performance comes at the price of over-parametrization, data hungriness, and expensive training cost (Werbos, 1990; Brown et al., 2020; Teubert et al., 2021; Neyshabur et al., 2018; Marcus, 2018).

Original: Furthermore, even if sufﬁcient data is available, the resulting models learn representations of the input data that are highly specialized to the training task and difﬁcult to adapt in different contexts.
Revised: Furthermore, even if sufﬁcient data is available, the resulting models learn representations of the input data that are highly specialized to the training task and difﬁcult to adapt in different contexts.

Original: In addition, the remarkable performance of these methods is often the result of a substantial engineering effort and is not supported by theoretical results.
Revised: In addition, the remarkable performance of these methods is often the result of a substantial engineering effort and is not supported by theoretical results.

Original: Reservoir Computing (RC) (Schrauwen et al., 2007) offers an intriguing alternative strategy to cope with the limitations above, yet retaining the universal approximation properties typical of deep neural networks.
Revised: Reservoir Computing (RC) (Schrauwen et al., 2007) offers an intriguing alternative strategy to cope with the limitations above, yet retaining the universal approximation properties typical of deep neural networks.

Original: In RC, the learning is divided into two phases: ﬁrst, data is passed through an untrained reservoir which extracts a set of task-independent features; second, a simple and efﬁcient-to-train linear map (the readout map) projects such features into the desired output.
Revised: In RC, the learning is divided into two phases: ﬁrst, data is passed through an untrained reservoir which extracts a set of task-independent features; second, a simple and efﬁcient-to-train linear map (the readout map) projects such features into the desired output.

Original: The critical point is that the design of the reservoir determines the expressiveness of the features, and several alternatives can be found in the literature (see (Gauthier et al., 2021) and references therein).
Revised: The critical point is that the design of the reservoir determines the expressiveness of the features, and several alternatives can be found in the literature (see (Gauthier et al., 2021) and references therein).

Original: A powerful reservoir is offered by the Signature Transform , often simply referred to as Signature, stemming from rough path theory (Ben Hambly, 2010;
Revised: A powerful reservoir is offered by the Signature Transform , often simply referred to as Signature, stemming from rough path theory (Ben Hambly, 2010;

Original: Friz and Hairer, 2020).
Revised: Friz and Hairer, 2020).

Original: The Signature of a path consists in enhancing the path with additional curves, which, in the smooth case, corresponds to iterated integrals of the curve with itself.
Revised: The Signature of a path consists in enhancing the path with additional curves, which, in the smooth case, corresponds to iterated integrals of the curve with itself.

Original: A profound mathematical result (Levin et al., 2016) guarantees that the solution of a (rough) differential equation can be approximated arbitrarily wellby a linear map of the Signature of the input signals (a.k.a. controls).
Revised: A profound mathematical result (Levin et al., 2013) guarantees that the solution of a (rough) differential equation can be approximated arbitrarily well

Original: In the machine learning jargon, the Signature of a path can be interpreted as a feature map extracting all its geometrical properties

Original: [Figure]
Revised: [Figure]

Original:  and thus allowing a simple linear map to approximate any function of it.
Revised: by a linear map of the Signature of the input signals (a.k.a. controls). In the machine learning jargon, the Signature of a path can be interpreted as a feature map extracting all its geometrical properties and thus allowing a simple linear map to approximate any function of it.

Original: On the other hand, it is often the case that the reservoir features are very high dimensional, and hence are particularly expensive to calculate and use in downstream tasks.
Revised: On the other hand, it is often the case that the reservoir features are very high dimensional, and hence are particularly expensive to calculate and use in downstream tasks.

Original: In addition, the high-dimensionality of the Signature reservoir poses additional challenges for modern gradient-based optimizers as convergence rates suffer from a linear dependence in the model dimension (Bottou et al., 2018).
Revised: In addition, the high-dimensionality of the Signature reservoir poses additional challenges for modern gradient-based optimizers as convergence rates suffer from a linear dependence in the model dimension (Bottou et al., 2018).

Original: Inspired by the remarkable theoretical properties of the Signature reservoir and motivated to ﬁx its practical pitfalls, our contribution is to showcase the effectiveness of Randomized Signatures (Cuchiero et al., 2021b;a), a recently introduced reservoir of random features.
Revised: Inspired by the remarkable theoretical properties of the Signature reservoir and motivated to ﬁx its practical pitfalls, our contribution is to showcase the effectiveness of Randomized Signatures (Cuchiero et al., 2021b;a), a recently introduced reservoir of random features.

Original: These features provably hold the same geometric properties and approximation power as the Signature, yet are often more efﬁcient to compute and of lower dimensionality.
Revised: These features provably hold the same geometric properties and approximation power as the Signature, yet are often more efﬁcient to compute and of lower dimensionality.

Original: To extract Randomized Signatures, we numerically integrate a set of random linear stochastic differential equations driven by the path itself.
Revised: To extract Randomized Signatures, we numerically integrate a set of random linear stochastic differential equations driven by the path itself.

Original: We assess the expressiveness of the proposed random features on several tasks, including nonparametric (black-box) system identiﬁcation problems arising from complex controlled nonlinear dynamical systems as well as time series classiﬁcation.
Revised: We assess the expressiveness of the proposed random features on several tasks, including nonparametric (black-box) system identiﬁcation problems arising from complex controlled nonlinear dynamical systems as well as time series classiﬁcation.

Original: The paper is structured as follows:
Revised: The paper is structured as follows:

Original: In Section 2, we link the Randomized Signature approach with related research areas in the literature; in Section 3, we present relevant theoretical results motivating our method; in Section 4, we assess the effectiveness of Randomized Signatures on various system identiﬁcation and time series classiﬁcation tasks, showing promising results.
Revised: In Section 2, we link the Randomized Signature approach with related research areas in the literature; in Section 3, we present relevant theoretical results motivating our method; in Section 4, we assess the effectiveness of Randomized Signatures on various system identiﬁcation and time series classiﬁcation tasks, showing promising results when compared to other signature approaches for reservoir computing.

Original: Finally, in Section 5, we discuss our conclusions and possible future developments.
Revised: Finally, in Section 5, we discuss our conclusions and possible future developments.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Random Features and Reservoir Computing.
Revised: Random Features and Reservoir Computing.

Original: The idea of extracting features based on random operations is not new and has seen a number of successful applications over the past years.
Revised: The idea of extracting features based on random operations is not new and has seen a number of successful applications over the past years.

Original: Of particular note, the seminal work of Rahimi and Recht (2008) proposes to accelerate kernel machines by learning random features whose inner product matches that of a target shift-invariant kernel.
Revised: Of particular note, the seminal work of Rahimi and Recht (2008) proposes to accelerate kernel machines by learning random features whose inner product matches that of a target shift-invariant kernel.

Original: The trade-off between generalization and computational efﬁciency of learning with random features has then been rigorously studied by Rudi and Rosasco (2017).
Revised: The trade-off between generalization and computational efﬁciency of learning with random features has then been rigorously studied by Rudi and Rosasco (2017).

Original: A conceptually very similar rationale is introduced by a parallel series of works exploring the topic of Reservoir Computing (Schrauwen et al., 2007).
Revised: A conceptually very similar rationale is introduced by a parallel series of works exploring the topic of Reservoir Computing (Schrauwen et al., 2007).

Original: Similarly to us, Echo State Networks (Jaeger, 2003) evolve the input state by a series of ﬁxed random projection (the reservoir) and generate the output by applying a trainable linear projection over the hidden states.
Revised: Similarly to us, Echo State Networks (Jaeger, 2003) evolve the input state by a series of ﬁxed random projection (the reservoir) and generate the output by applying a trainable linear projection over the hidden states.

Original: However, we make the additional step of linking the random features to the Signature of the input path and, as shown below, the evolution of the features is dictated by a randomly-evolved stochastic differential equation driven by the input path.
Revised: However, we make the additional step of linking the random features to the Signature of the input path and, as shown below, the evolution of the features is dictated by a randomly-evolved stochastic differential equation driven by the input path.

Original: Controlled Differential Equations.
Revised: Controlled Differential Equations.

Original: Our work is also related with a series of recent papers investigating the problems of how to process irregular time series and to condition a model on incoming information through the lens of controlled differential equations (CDEs) (Kidger et al., 2020; Morrill et al., 2021).
Revised: Our work is also related with a series of recent papers investigating the problems of how to process irregular time series and to condition a model on incoming information through the lens of controlled differential equations (CDEs) (Kidger et al., 2020; Morrill et al., 2021).

Original: Interestingly, Kidger et al.
Revised: Interestingly, Kidger et al.

Original: (2020) show that the action of a linear layer on the ﬁnal output of a trained Neural CDEs – the extension of Neural ODEs (Chen et al., 2019) to CDEs – results in a universal function approximator.
Revised: (2020) show that the action of a linear layer on the ﬁnal output of a trained Neural CDEs – the extension of Neural ODEs (Chen et al., 2018) to CDEs – results in a universal function approximator.

Original: Differently from them, our method is less computationally expensive as the only parameters we need to train are those of the ﬁnal linear readout layer.
Revised: Differently from them, our method is less computationally expensive as the only parameters we need to train are those of the ﬁnal linear readout layer.

Revised: Fundamentally different to our work, the approach of Morrill et al.

Revised: ﬁrst randomly projects the possibly high dimensional controls into a lower dimensional space and then extracts the Truncated

Revised: Signature from such compressed input.

Revised: Our method instead extracts a random compression of the signature of the original input controls.

Original: Rough Path Theory.
Revised: Rough Path Theory.

Original: Rough path theory is about describing how possibly highly oscillatory (rough) control path interact with nonlinear systems (Lyons, 2014).
Revised: Rough path theory is about describing how possibly highly oscillatory (rough) control path interact with nonlinear systems (Lyons, 2014).

Original: The concept of Signature is introduced in this context to provide a powerful description of the input path, removing the need to look at the ﬁne structure of the path itself.
Revised: The concept of Signature is introduced in this context to provide a powerful description of the input path, removing the need to look at the ﬁne structure of the path itself.

Original: Recent years have seen a resurgence of these ideas, which have been revisited from a machine learning perspective (Bonnier et al., 2019; Kidger and Lyons, 2020).
Revised: Recent years have seen a resurgence of these ideas, which have been revisited from a machine learning perspective (Bonnier et al., 2019; Kidger and Lyons, 2021).

Original: Our analysis is strongly inﬂuenced by the work of Cuchiero et al.
Revised: Our analysis is strongly inﬂuenced by the work of Cuchiero et al.

Original: (2021a;b), who, starting from the observation that the Signature is an inﬁnite dimensional reservoir, establishes that its information content can be efﬁciently compressed by a random projection performed by a dynamical system driven by random vector ﬁelds.
Revised: (2021a;b), who, starting from the observation that the Signature is an inﬁnite dimensional reservoir, establishes that its information content can be efﬁciently compressed by a random projection performed by a dynamical system driven by random vector ﬁelds.

Original: As we show later, this results in a random reservoir which preserves the properties of the Signature while living in a ﬁnite dimensional space.
Revised: As we show later, this results in a random reservoir which preserves the properties of the Signature while living in a ﬁnite dimensional space.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We provide the theoretical tools supporting the Randomized Signature approach for feature extraction in times series.
Revised: We provide the theoretical tools supporting the Randomized Signature approach for feature extraction in times series.

Original: A formal discussion can be found in (Friz and Hairer, 2020) and (Cuchiero et al., 2021b).
Revised: A formal discussion can be found in (Friz and Hairer, 2020) and (Cuchiero et al., 2021b).

Original: 3. R ANDOMIZED S IGNATURE OF A P ATH
Revised: 3. R ANDOMIZED S IGNATURE OF A P ATH

Original: Let X : [0 , T ] → R d be a continuous piecewise smooth d -dimensional path X
Revised: Let X : [0 , T ] → R d be a continuous piecewise smooth d -dimensional path X

Original: = (cid:0) X 1 , · · · , X d (cid:1) .
Revised: = (cid:0) X 1 , · · · , X d (cid:1) .

Original: We will refer to X as the control and to its single components
Revised: We will refer to X as the control and to its single components

Original: X i as controls .
Revised: X i as controls .

Original: We denote by { e 1 , . . .
Revised: We denote by { e 1 , . . .

Original: , e d } the canonical basis of R d .
Revised: , e d } the canonical basis of R d .

Original: Deﬁnition 1 (Signature)
Revised: Deﬁnition 1 (Signature)

Original: For any t ∈ [0 , T ] , the Signature of a continuous piecewise smooth path
Revised: For any t ∈ [0 , T ] , the Signature of a continuous piecewise smooth path

Original: X : [0 , T ] → R d on [0 , t ] is the countable collection S t := (cid:0) 1 , S 1 t , S 2 t , . . .
Revised: X : [0 , T ] → R d on [0 , t ] is the countable collection S t := (cid:0) 1 , S 1 t , S 2 t , . . .

Original: (cid:1) ∈ (cid:81) ∞
Revised: (cid:1) ∈ (cid:81) ∞

Original: k =0 (cid:0) R d (cid:1) ⊗ k where, for each k ≥ 1 , the entries S kt are the iterated integrals deﬁned as
Revised: k =0 (cid:0) R d (cid:1) ⊗ k where, for each k ≥ 1 , the entries S kt are the iterated integrals deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the Truncated Signature of X of order M ≥ 0 as
Revised: We deﬁne the Truncated Signature of X of order M ≥ 0 as

Original: [Equation]
Revised: [Equation]

Original: [-Remark 
-]
Revised: [-Remark 
-]

Original: We highlight that S kt lives in (cid:0) R d (cid:1) ⊗ k , which is the space of tensors of shape ( d, . . . , d ) given by R d ⊗ · · · ⊗ R d for k times. Practical example ofthe computation of Signatures (one dimension). To enhance intuition, we exemplify the computation of Signatures  in the simplest setting possible.
Revised: A practical example of computation of Signatures is presented in Appendix A.3.

Original: Let X : [0 , T ] → R , then S 1 t = (cid:82) t 0 dX s — which is exactly X t − X 0 .

Original: To get S 2 t , we instead have to compute the following iterated integral:

Original: S 2 t = (cid:82) t 0 (cid:0)(cid:82) v 0 dX s (cid:1) dX v .

Original: Signatures of higher order S jt are computed in a similar way, by iteratively integrating the path j times.

Original: As a practical example, let X t = t . Then it is easy to see that S jt = t j j ! . Now let Y ( t ) be an analytic function of time for which we have Y
Revised: The deﬁnition in the last paragraph suggests that the Signatures of X can be used to approximate any regular enough function of X , for instance the solution to differential equations controlled by

Original: [Equation]

Original: Taylor’s theorem combined with the previous computation of the Signatures implies that Y can be approximated as a linear map of the Signatures of t .

Original: Finally, note that S j t in this case gets smaller and smaller in magnitude as j increases. This suggests that the truncated Signature can be safely used to approximate Y . The computations presented in the last paragraph suggests that the Signatures of X can be used to approximate any regular enough function of X , for instance the solution to differential equations controlled by X .
Revised: X .

Original: The following result makes this argument precise in the multidimensional setting.
Revised: The following result makes this argument precise in the multidimensional setting.

Original: Theorem 1 (Signature is a Reservoir)
Revised: Theorem 1 (Signature is a Reservoir)

Original: Let V i : R m → R m , i = 1 , . . .
Revised: Let V i : R m → R m , i = 1 , . . .

Original: , d be vector ﬁelds regular enough s.t. dY t
Revised: , d be vector ﬁelds regular enough s.t. dY t

Original: =
Revised: =

Original: (cid:80) di
Revised: (cid:80) di

Original: =1 V i
Revised: =1 V i

Original: ( Y t )
Revised: ( Y t )

Original: dX it , Y 0 =
Revised: dX it , Y 0 =

Original: y ∈ R m , admits a unique solution Y t : [0 , T ] → R m .
Revised: y ∈ R m , admits a unique solution Y t : [0 , T ] → R m .

Original: Then, for any smooth test function F : R m → R and for every M ≥ 

Revised: Then, for any smooth test function F : R m → R and for every M ≥

Original: there is a time-homogeneous linear operator L : T M (cid:0) R d 
Revised: there is a time-homogeneous linear operator L : T M (cid:0) R d (cid:1)

Original: [Equation]

Original: → R which depends only on ( V 1 , . .
Revised: → R which depends only on ( V 1 , . .

Original: , V d , F, M, y ) s.t.
Revised: , V d , F, M, y ) s.t.

Original: [Equation]
Revised: [Equation]

Original: This theorem suggests the ﬁrst M entries of the Signature of X are sufﬁcient to linearly explain the solution of any differential equation driven by it.
Revised: This theorem suggests the ﬁrst M entries of the Signature of X are sufﬁcient to linearly explain the solution of any differential equation driven by it.

Original: In addition to this, it supports the claim that such features are valuable for any downstream usage.
Revised: In addition to this, it supports the claim that such features are valuable for any downstream usage.

Original: Computational Complexity and dimensionality of Signatures.

Original: The computational complexity for computing S Mt is O

Original: ( d M ) .

Original: Indeed, consider d = 2 : S 2 t is a 2 × 2 matrix with elements (cid:82) t 0 (cid:0)(cid:82) v 0 dX 1 s (cid:1) dX 1 v , (cid:82) t 0 (cid:0)(cid:82) v 0 dX 1 s (cid:1) dX 2 v , (cid:82) t 0 (cid:0)(cid:82) v 0 dX 2 s (cid:1) dX 1 v

Original: and (cid:82) t 0 (cid:0)(cid:82) v 0 dX 2 s (cid:1) dX 2 v .

Original: For M = 3 , the object to compute is instead a 2 × 2 × 2 tensor, containing all integrals of the type (cid:82) t 0 (cid:0)(cid:82) w 0 (cid:0)(cid:82) v 0 dX i 1 s (cid:1) dX i 2 v (cid:1) dX i 3 w for all i 1 , i 2 , i 3 ∈ { 1 , 2 } .

Original: Hence, the complexity — as well as the features dimensionality — scales exponentially in M .

Original: 3. R ANDOMIZED S IGNATURE OF A P ATH AND ITS THEORETICAL GUARANTEES
Revised: 3. R ANDOMIZED S IGNATURE OF A P ATH AND ITS THEORETICAL GUARANTEES

Original: Calculating S Mt requires the calculation of d M +1 − 1 d −iterated integrals  – which in total quickly becomes computationally expensive.
Revised: Calculating S Mt requires the calculation of d M +1 − 1 d −iterated integrals (Appendix A.3) – which in total quickly becomes computationally expensive.

Original: Of course several techniques have been developed to circumvent this problem,in particular kernelization techniques, see, e.g., Kidger and Lyons (2020).
Revised: Several computational techniques have been developed to circumvent this problem, see, e.g. (Kidger and Lyons, 2021).

Original: The next result provides a practical description of how it is possible to reduce the computational burden without losing too much explanatory power .
Revised: The next result provides a practical description of how it is possible to reduce the computational burden without losing too much explanatory power .

Original: As such, this results provides the theoretical foundation for our approach.
Revised: As such, this results provides the theoretical foundation for our approach.

Original: Formal statements are provided in Appendix.

Original: Theorem 2 (Randomized Signature (Informal))
Revised: Theorem 2 (Randomized Signature (Informal))

Original: For any k ∈ N big enough and appropriately chosen random matrices
Revised: For any k ∈ N big enough and appropriately chosen random matrices

Original: A 1 , . . . ,
Revised: A 1 , . . . ,

Original: A d in R k × k , random shifts b 1 , . . .
Revised: A d in R k × k , random shifts b 1 , . . .

Original: , b d in R k × 1 , random starting point z in R k × 1 , and any ﬁxed activation function σ , the solution of
Revised: , b d in R k × 1 , random starting point z in R k × 1 , and any ﬁxed activation function σ , the solution of

Original: [Equation]
Revised: [Equation]

Original: – called the Randomized Signature of X – has comparable approximation power as the Signature itself and maintains its geometric properties.
Revised: – called the Randomized Signature of X – has comparable approximation power as the Signature itself and maintains its geometric properties

Revised: The formal statement of such a result is the combination of Theorem 8, Theorem 9, and Deﬁnition 10 (see Appendix).

Revised: A complete formal discussion as well as the original statement of this theorem can be found in the paper (Cuchiero et al., 2021a).

Revised: In a nutshell, the Randomized Signature Z follows a dynamics which provides an efﬁcient and powerful compression of the Signature through a low-dimensional random projection.

Revised: The expressiveness of such projection is guaranteed by a standard Johnson-Lindenstrauss argument (Cuchiero et al., 2021a).

Original: Computational complexity and dimensionality of Randomized Signatures.
Revised: Computational complexity and dimensionality of Randomized Signatures.

Original: The computational complexity of calculating Z t is O ( k 2 d ) , while its dimensionality is O ( kd ) .
Revised: The computational complexity of calculating Z t is O ( k 2 d ) , while its dimensionality is O ( k ) .

Original: However, we should then ask the crucial question “ how should we compare k with M ? ”.
Revised: However, we should then ask the crucial question “ how should we compare k with M ? ”.

Original: We show experimentally in Section 4 that, in order to match the approximation guarantees of the Truncated Signature of order M , the number of required Randomized Signatures k is not too big — in particular it is not exponential in M .
Revised: We show experimentally in Section 4 that, in order to match the approximation guarantees of the Truncated Signature of order M , the number of required Randomized Signatures k is not too big — in particular it is not exponential in M .

Original: This conﬁrms that working with Randomized Signatures is often less computationally demanding and results in lower-dimensional features.
Revised: This conﬁrms that working with Randomized Signatures is often less computationally demanding and results in lower-dimensional features.

Original: We conclude the section remarking the fascinating implications of choosing a random A i s and b i s.
Revised: We conclude the section remarking the fascinating implications of choosing a random A i s and b i s.

Original: Theorem 3 (Density of Randomized Signatures (Informal))
Revised: Theorem 3 (Density of Randomized Signatures (Informal))

Original: For any sequences of time points 0 ≤ t 0 < . . .
Revised: For any sequences of time points 0 ≤ t 0 < . . .

Original: < t N ≤ T and points z t 0 , . .
Revised: < t N ≤ T and points z t 0 , . .

Original: z t N ∈ R k we can ﬁnd a smooth control X such that the solution Z of Equation 4 at time t i is such that Z t i =
Revised: z t N ∈ R k we can ﬁnd a smooth control X such that the solution Z of Equation 4 at time t i is such that Z t i =

Original: z t i , for i = 0 , . . .
Revised: z t i , for i = 0 , . . .

Original: , N .
Revised: , N .

Original: Additionally, if the control X is a d -dimensional Brownian motion, then the solution of Equation 4 at any point in time t > 0 admits a smooth density with respect to Lebesgue measure.
Revised: Additionally, if the control X is a d -dimensional Brownian motion, then the solution of Equation 4 at any point in time t > 0 admits a smooth density with respect to Lebesgue measure.

Original:  T HE E XPRESSIVE P OWER OF R ANDOMIZED S IGNATURES
Revised:  T HE E XPRESSIVE P OWER OF R ANDOMIZED S IGNATURES

Original: Algorithm 1 Generate Randomized Signature
Revised: To compute the Randomized Signature, we rely on an Euler-Maruyama approximation; details are given in Algorithm 1. To start, we experimentally validate some of the results presented above. In

Original: Require:
Revised: Require:

Original: X ∈ R d sampled at 0 = t 0 < · · · < t N = T , Randomized Signature dimension k , activation function σ .
Revised: X ∈ R d sampled at 0 = t 0 < · · · < t N = T , Randomized Signature dimension k , activation function σ .

Original: Initialize: Z 0 ∈ R k , A i ∈ R k × k , b i ∈ R k to have iid standard normal entries for nin { 1 ..N } do
Revised: Initialize n do

Original: [Equation]
Revised: [Equation]

Original: end for

Original: To compute the Randomized Signature, we rely on an Euler-Maruyama approximation; details are given in Algorithm 1.

Original: To start, we experimentally validate some of the results presented above.

Original: In particular, we show that the Truncated Signature of a control lies in the linear span of its Randomized
Revised: for particular, we show that the Truncated Signature of a control lies in the linear span of its Randomized

Original: Signatures, thus validating Theorem 2.
Revised: Signatures, thus validating Theorem 2.

Original: As a second proof of concept, we reconstruct a control X as a linear combination of its Randomized Signatures, showcasing the expressiveness of such random features .
Revised: As a second proof of concept, we reconstruct a control X as a linear combination of its Randomized Signatures, showcasing the expressiveness of such random features .

Original: Finally, we combine Theorem 1 and Theorem 2 and we use Z to learn the dynamics of complex nonlinear systems of differential equations and to perform time series classiﬁcation.
Revised: Finally, we combine Theorem 1 and Theorem 2 and we use Z to learn the dynamics of complex nonlinear systems of differential equations and to perform time series classiﬁcation.

Original: Our result show that the Randomized Signature is a powerful random reservoir which guarantees — both in theory and practice — the same geometrical properties and approximation power as the
Revised: Our result show that the Randomized Signature is a powerful random reservoir which guarantees — both in theory and practice — the same geometrical properties and approximation power as the

Original: Signature itself, while often being of lower dimensionality and less computationally expensive both at extraction and at deployment time. 
Revised: Signature itself, while often being of lower dimensionality and less computationally expensive both at extraction and at deployment time. Note on the choice of σ .

Revised: While the choice of σ does not affect the theoretical results (Cuchiero et al., 2021a;b), we found that a careful choice is needed for optimal results.

Revised: Inspired by seminal works on the stability of deep linear networks (Glorot and Bengio, 2010) and by the connection to neural ODEs (Chen et al., 2018), we selected σ to be linear, with slope 1 d ×√ k .

Revised: This choice, complemented with additional ﬁne-tuning, leads to the best performance for downstream tasks.

Original: 4. L EARNING THE S IGNATURES
Revised: 4. L EARNING THE S IGNATURES

Original: In this experiment, we show that we can approximate the Truncated Signature of a control using a linear combination of its Randomized Signatures.
Revised: In this experiment, we show that we can approximate the Truncated Signature of a control using a linear combination of its Randomized Signatures.

Original: We ﬁx an equally-spaced partition D
Revised: We ﬁx an equally-spaced partition D

Original: = { t 0 , · · · , t N } of [0 , 1] and calculate the truncated Signature of order M of a d -dimensional path X on D , i.e. S
Revised: = { t 0 , · · · , t N } of [0 , 1] and calculate the truncated Signature of order M of a d -dimensional path X on D , i.e. S

Original: Mt i , for i = 0 , · · · , N .
Revised: Mt i , for i = 0 , · · · , N .

Original: For a ﬁxed value of k , we use Algorithm 1 to generate a realization of the Randomized Signature Z of X on D .
Revised: For a ﬁxed value of k , we use Algorithm 1 to generate a realization of the Randomized Signature Z of X on D .

Original: For sake of clarity, the truncated Signatures on D are reshaped into an (cid:0) N × (cid:0)(cid:0) d ( M +1) − 1 (cid:1) /
Revised: For sake of clarity, the truncated Signatures on D are reshaped into an (cid:0) N × (cid:0)(cid:0) d ( M +1) − 1 (cid:1) /

Original: ( d − 1) − 1 (cid:1)(cid:1) -dimensional matrix called S trueM while the Randomized Signatures on D is represented as an ( N × k ) -dimensional matrix called Z Randk .
Revised: ( d − 1) − 1 (cid:1)(cid:1) -dimensional matrix called S trueM while the Randomized Signatures on D is represented as an ( N × k ) -dimensional matrix called Z Randk .

Original: Finally, we train a Ridge Regression to ﬁnd a parameter matrix W ∈ R k × (( d ( M +1) − 1 ) /
Revised: Finally, we train a Ridge Regression to ﬁnd a parameter matrix W ∈ R k × (( d ( M +1) − 1 ) /

Original: ( d − 1) − 1 ) mapping the Randomized Signatures Z Randk into the truncated Signatures S trueM .
Revised: ( d − 1) − 1 ) mapping the Randomized Signatures Z Randk into the truncated Signatures S trueM .

Original: More precisely, we consider a 10 -dimensional control X =
Revised: More precisely, we consider a 10 -dimensional control X =

Original: ( t, W t ) where W is a 9 -dimensional
Revised: ( t, W t ) where W is a 9 -dimensional

Original: Brownian motion with independent components, and ﬁx M = 6 .
Revised: Brownian motion with independent components, and ﬁx M = 6 .

Original: For each k ∈ { 1 , · · · , 200 } , we generate Z Randk , perform a simple linear regression and calculate the L 2 approximation error.
Revised: For each k ∈ { 1 , · · · , 200 } , we generate Z Rand k , perform a simple linear regression and calculate the L 2 approximation error.

Original: Results in Figure 3 indicate that the approximation error decreases as k increases.
Revised: Results in Figure 3 indicate that the approximation error decreases as k increases.

Original: This conﬁrms that, as ensured by Theorem 2, the random features constructed as prescribed by Equation 4 hold the same information content as the Signature and thus maintain its approximation power and geometric properties.
Revised: This conﬁrms that, as ensured by Theorem 2, the random features constructed as prescribed by Equation 4 hold the same information content as the Signature and thus maintain its approximation power and geometric properties.

Original: Finally, we highlight that, if we were to be satisﬁed with an approximation error of order 10 − 4 , we could achieve it with roughly with k = 190 Randomized Signatures.
Revised: Finally, we highlight that, if we were to be satisﬁed with an approximation error of order 10 − 4 , we could achieve it with roughly with k = 190 Randomized Signatures.

Original: To conclude, this means that, instead of calculating (cid:0)(cid:0) d
Revised: To conclude, this means that, instead of calculating (cid:0)(cid:0) d

Original: ( M +1) − 1 (cid:1) /
Revised: ( M +1) − 1 (cid:1) /

Original: ( d − 1) − 1 (cid:1) =
Revised: ( d − 1) − 1 (cid:1) =

Original: 1111110 integrals per time step, we could just perform k 2 d = 36100 calculations per time step — which is 3 times cheaper .
Revised: 1111110 integrals per time step, we could just perform k 2 d = 36100 calculations per time step — which is 3 times cheaper .

Revised: More remarkably, this high feature dimensionality ( 1111110 for Signature) makes any model for downstream tasks exponentially more complex than the that based on Randomized Signature (only 190 features).

Original: Modeling choices and details are reported in Appendix B.2.[Figure] [Figure] ̂
Revised: Modeling choices and details are reported in Appendix B.2.

Original: [Figure]

Original: 4. R ECONSTRUCTION AND R ANDOMIZED S IGNATURE A UTOENCODERS
Revised: 4. R ECONSTRUCTION AND R ANDOMIZED S IGNATURE A UTOENCODERS

Original: The Randomized Signature of X is rich enough to reconstruct it almost perfectly.
Revised: The Randomized Signature of X is rich enough to reconstruct it almost perfectly.

Original: In practice, we consider an “autoencoder” architecture, whose encoder is our random feature generator and the decoder is a trained linear projection mapping the Randomized Signatures into the original path.
Revised: In practice, we consider an “autoencoder” architecture, whose encoder is our random feature generator and the decoder is a trained linear projection mapping the Randomized Signatures into the original path.

Original: For this experiment, we ﬁx an equally-spaced partition D
Revised: For this experiment, we ﬁx an equally-spaced partition D

Original: = { t 0 , · · · , t N } and consider a single realization
Revised: = { t 0 , · · · , t N } and consider a single [Figure] [Figure] [Figure]

Original: X Train ofan Ornstein–Uhlenbeck process.
Revised: [Figure] [Figure] V a l u e Out of

Original:  Ultimately, we train a Ridge Regressor to map the Randomized Signature Z of the one and only train sample X Train into X Train itself.
Revised: [Figure] Extracted X t True X t X t ̂ X t Z t realization X Train of an Ornstein–Uhlenbeck process. Ultimately, we train a Ridge Regressor to map the Randomized Signature Z of the one and only train sample X Train into X Train itself.

Original: Table 5 in Appendix shows the average reconstruction error on N Test = 10000 test samples as we repeat the experiment for different choices of k .
Revised: Table 6 in Appendix shows the average reconstruction error on N Test = 10000 test samples as we repeat the experiment for different choices of k .

Original: Such results are in accordance with Figure 4 which shows an example of the true and generated time series of a test sample.
Revised: Such results are in accordance with Figure 4 which shows an example of the true and generated time series of a test sample.

Original: This conﬁrms that the Randomized
Revised: This conﬁrms that the Randomized

Original: Signature of a path holds an equivalent information content as the path itself while being more usable for any downstream tasks.
Revised: Signature of a path holds an equivalent information content as the path itself while being more usable for any downstream tasks.

Original: Modeling choices and details are reported in Appendix B.2.
Revised: Modeling choices and details are reported in Appendix B.2.

Original: 4. L EARNING SDE SOLUTIONS WITH A LINEAR MAP ON THE R ANDOMIZED S IGNATURE
Revised: 4. L EARNING SDE SOLUTIONS WITH A LINEAR MAP ON THE R ANDOMIZED S IGNATURE

Original: In this series of experiments, we provide evidence of the potential of Randomized Signatures in several downstream tasks.
Revised: In this series of experiments, we provide evidence of the potential of Randomized Signatures in several downstream tasks.

Original: First of all, as shown in Appendix B, we prove that our approach can accurately learn the dynamics of several rough differential equations: we highlight that the algorithm has only access to the input controls and the output response of the system but it is completely agnostic with respect to the law of the dynamics itself .
Revised: First of all, as shown in Appendix B, we prove that our approach can accurately learn the dynamics of several rough differential equations: we highlight that the algorithm has only access to the input controls and the output response of the system but it is completely agnostic with respect to the law of the dynamics itself .

Original: To start with, we focus on the nonlinear dynamics of 1 -dimensional stochastic
Revised: To start with, we focus on the nonlinear dynamics of 1 -dimensional stochastic

Original: Double Well.
Revised: Double Well.

Original: Secondly, we study the multidimensional case of the 4 -dimensional Ornstein–Uhlenbeck.
Revised: Secondly, we study the multidimensional case of the 4 -dimensional Ornstein–Uhlenbeck.

Original: Then, we investigate the performance on the extremely rough dynamics of Fractional Geometric Brownian Motion even for low values of the Hurst parameter H .
Revised: Then, we investigate the performance on the extremely rough dynamics of Fractional Geometric Brownian Motion even for low values of the Hurst parameter H .

Original: Additionally, we obtain promising results in learning the dynamics of an Enzyme-Substrate
Revised: Additionally, we obtain promising results in learning the dynamics of an Enzyme-Substrate

Original: Reactions (Ingalls, 2013).
Revised: Reactions (Ingalls, 2013).

Original: These preliminary results are presented in the Appendix.

Original: Leveraging these results, we conﬁdently implement three more experiments: the ﬁrst demonstrates that this approach has sustainable performances even when the input control is irregularly sampled. The following two are highly nonlinear dynamical systems with a real world application. 1 -Dimensional Stochastic Double Well - Irregularly Sampled Time Grid. Let us recall that the dynamics of the 1 -Dimensional Stochastic Double Well process is given by
Revised: We also demonstrate that this approach has sustainable performance even when the input control is irregularly sampled as well as Neutral Controlled Differential Equations.

Original: [Equation]

Original: where W t is a 1 -dimensional Brownian motion, and ( µ, θ, σ ) ∈ R × R + × R + . For each train and test sample, the partition D of [0 , 1] is made of N randomly drawn times. More precisely, D = { 0 , t 1 , · · · , t N − 1 , 1 } such that t k = 1 / (1 − exp( − s k )) and { s 1 , · · · , s N − 1 } are N − 2 independent realizations of a uniform distribution U [0 , 1] sorted increasingly. As a result, the probability that two samples share the same D is null.
Revised: Due to space constraints, these results are presented in the Appendix.

Original: We train a Ridge Regressor with N Train = 10000 train samples and Figure 10 shows the comparison of an out-of-sample generated and true trajectory. Finally, Tableshows the Relative L 2 Error on 10000 test samples as we vary the number of time steps N and k and while comparing it to the respective experiment in case the time grid is regularly spaced. As we can see, even though the performance are worse than the regularly sampled setup, this technique proves to be anyway reliable on irregularly sampled regimes.
Revised: Here, we show the performance in learning highly nonlinear dynamical systems with a real world application.

Original: Details are reported in Appendix B.2.

Original: Tumor Growth Kinetics.
Revised: Tumor Growth Kinetics.

Original: In this experiment, we consider predicting the output of a complex controlled nonlinear system using Randomized Signatures.
Revised: In this experiment, we consider predicting the output of a complex controlled nonlinear system using Randomized Signatures.

Original: The Simeoni Tumor model (Simeoni

Original: ( N, k )

Original: = (11 , 111) ( N, k ) =

Original: (101 , 222) ( N, k ) =

Original: Irregular 0 .0 .0 .

Original: Regular 0 .0 .0 . [Figure] et al., 2004) is widely used to predict the evolution of tumor weight Y t under the concentration of a treatment drug X t .
Revised: The Simeoni Tumor model (Simeoni et al., 2004) is widely used to predict the evolution of tumor weight Y t under the concentration of a treatment drug X t .

Original: The system is speciﬁed by the following equations
Revised: The system is speciﬁed by the following equations

Original: [Equation]
Revised: [Equation]

Original: We ﬁx an equally spaced partition D of [0 , 10] and perform Ridge Regression to map 10000 instances of the Randomized Signature of the controls into the respective integrated solution Y t .
Revised: We ﬁx an equally spaced partition D of [0 , 10] and perform Ridge Regression to map 10000 instances of the Randomized Signature of the controls into the respective integrated solution Y t .

Original: We sample X t from the law of a Scaled Squared Brownian motion 0 .
Revised: We sample X t from the law of a Scaled Squared Brownian motion 0 .

Original: 25 · W 2 t where W t is a 1 -dimensional Brownian
Revised: 25 · W 2 t where W t is a 1 -dimensional Brownian

Original: Motion (BM) and the left side of Figure 5 shows the comparison of the true and the generated time series on a test sample along with the respective path of the control.
Revised: Motion (BM) and the left side of Figure 5 shows the comparison of the true and the generated time series on a test sample along with the respective path of the control.

Original: To test the robustness of our approach, we use controls that do not follow the law used in training .
Revised: To test the robustness of our approach, we use controls that do not follow the law used in training .

Original: More precisely, the middle of Fig.
Revised: More precisely, the middle of Fig.

Original: 5 shows what happens if we use 1 { W 2 t > 1 } as input control instead of 0 .
Revised: 5 shows what happens if we use 1 { W 2 t > 1 } as input control instead of 0 .

Original: 25 · W 2 t while its right does the same as we use W t .
Revised: 25 · W 2 t while its right does the same as we use W t .

Original: These two experiments prove that the algorithm learnt much more than merely predicting the behaviour of the system under ordinary circumstances.
Revised: These two experiments prove that the algorithm learnt much more than merely predicting the behaviour of the system under ordinary circumstances.

Original: The ﬁrst shows that the algorithm has “understood” that the absence of drug leads to the growth of the tumor while it dies back once the medicine is injected.
Revised: The ﬁrst shows that the algorithm has “understood” that the absence of drug leads to the growth of the tumor while it dies back once the medicine is injected.

Original: On the other hand, the second suggests that the algorithm cannot behave properly when the input becomes negative — not allowed by the problem deﬁnition (negative drug concentration).
Revised: On the other hand, the second suggests that the algorithm cannot behave properly when the input becomes negative — not allowed by the problem deﬁnition (negative drug concentration).

Original: Modeling choices and details are reported in Appendix B.2.
Revised: Further details are reported in Appendix B.2.

Original: Electrochemical Battery Model.
Revised: Electrochemical Battery Model.

Original: In this experiment, we learn the dynamics of the electrochemical battery model proposed in (Daigle and Kulkarni, 2013) which returns the voltage Y as the current 
Revised: In this experiment, we learn the dynamics of the electrochemical battery model proposed in (Daigle and Kulkarni, 2013) which returns the voltage Y as the current [Figure] [Figure]

Original: X is drawn from the battery.
Revised: X is drawn from the battery.

Original: We use the open source NASA Prognostic Model Package (Teubert et al., 2021) to simulate voltage trajectories given input current control paths.
Revised: We use the open source NASA Prognostic Model Package (Teubert et al., 2021) to simulate voltage trajectories given input current control paths.

Original: On a ﬁxed equally spaced partition D of [0 , 500] , we model the input current with step functions taking values 0 or [Figure] on random sub-interval of [0 , 500] .
Revised: On a ﬁxed equally spaced partition D of [0 , 500] , we model the input current with step functions taking values 0 oron random sub-interval of [0 , 500] .

Original: To test the robustness of our approach, we train a Ridge Regression to map N Train = 1000 instances of k -dimensional Randomized Signature of the controls into the respective solutions to which we add white noise.
Revised: To test the robustness of our approach, we train a Ridge Regression to map N Train = 1000 instances of k -dimensional Randomized Signature of the controls into the respective solutions to which we add white noise.

Original: The left of Figure 6 shows the comparison between the ground truth and our prediction on an in-sample trajectory.
Revised: The left of Figure 6 shows the comparison between the ground truth and our prediction on an in-sample trajectory.

Original: In the same ﬁgure, we compare the generated time series for several choices of k on a test sample along with the path of the input control.
Revised: In the same ﬁgure, we compare the generated time series for several choices of k on a test sample along with the path of the input control.

Original: In Figure 6 we show the performance of our method in comparison with a nonlinear neural ARX (NNARX) — which is often used in the modern control theory/system identiﬁcation literature as a benchmarch (see e.g. (Masti and Bemporad, 2018)), given its strong theoretical guarantees (Schoukens and Ljung, 2019).
Revised: In Figure 6 we show the performance of our method in comparison with a nonlinear neural ARX (NNARX) — which is often used in the modern control theory/system identiﬁcation literature as a benchmarch (see e.g. (Masti and Bemporad, 2018)), given its strong theoretical guarantees (Schoukens and Ljung, 2019).

Original: Table 6 in Appendix reports the performances in terms of
Revised: Table 8 in Appendix reports the performances in terms of

Original: MSE on 1000 test samples for our models compared to two versions of NNARX: NNARX 22 which has around 1000 trainable parameters just like our model with k = 1000 , and NNARX 1000 which matches our best model in terms of MSE but has around 10 6 trainable parameters.
Revised: MSE on 1000 test samples for our models compared to two versions of NNARX: NNARX 22 which has around 1000 trainable parameters just like our model with k = 1000 , and NNARX 1000 which matches our best model in terms of MSE but has around 10 6 trainable parameters.

Original: Modeling choices and details are reported in Appendix B.2.
Revised: Details are reported in Appendix B.2.

Revised: Comparison with Neural Controlled Differential Equations.

Revised: In Appendix B, we benchmark the performance of our model with that of NCDEs (Kidger et al., 2020), a recently proposed competitive deep learning technique for controlled dynamical system modelling.

Revised: We test the two models on the task of learning the dynamics of an Ornstein-Uhlenbeck Process, and showcase the superior performance of our method both in terms of computational complexity and accuracy.

Revised: 4. C OMPARISON WITH T RUNCATED S IGNATURES

Revised: In this experiment, we learn the dynamics of the Fractional Ornstein–Uhlenbeck process using our

Revised: Randomized Signature model and benchmark our results against a Truncated Signature model of order M .

Revised: The computational cost of extracting the Truncated Signature of order M = 3 for d controls is O ( d 3 ) while for Randomized Signature it is O ( k 2 d ) .

Revised: To keep the computational cost of extracting features equal across models, we select k = d .

Revised: As a result, the number of features of the Signature-based model is O ( d 3 ) while that of our model is O ( d ) .

Revised: In the spirit of Reservoir

Revised: Computing, we choose both models to be linear.

Revised: Hence, the number of trainable parameters is equal to the number of features.

Revised: This implies that, the number of trainable parameters of our model is orders of magnitude smaller than that of our benchmark (left of Figure 7) — this severely affects the training time (see e.g. (Bottou et al., 2018)) .

Revised: Figure 7 shows that our training time is almost independent on d .

Revised: Instead, the training complexity of the Signature-model is exponential in d and the Out-of-Sample performance degenerates as the underlying optimization problem explodes in dimension.

Revised: This result clearly highlights the data-hungriness of Signature-based models when the number of dimensions is high.

Revised: To conclude, we emphasize that if d = 80 , the storage cost of the Truncated Signature of a path is 200 MB while its Randomized Signature is just 32 KB.

Revised: As a consequence, training the former on N Train = 10000 would take an unpractical amount of time and storage memory while we were able to train ours in 4 .

Revised: 8896 seconds using just 320 MB and attaining an Out-of-Sample Error on N Test = 10000 equal to 0 .

Revised: Modeling choices and details are reported in Appendix B.

Original: 4. T IME S ERIES C LASSIFICATION
Revised: 4. T IME S ERIES C LASSIFICATION

Original: In time series classiﬁcation (TSC) the goal is to map an input time series to the corresponding label.
Revised: In time series classiﬁcation (TSC) the goal is to map an input time series to the corresponding label.

Original: Given the complex high-dimensional nature of temporal signals, the vast majority of classiﬁcation algorithms aims at effectively ﬁnding expressive representations of the original time series and then maps them into the ﬁnal prediction.
Revised: Given the complex high-dimensional nature of temporal signals, the vast majority of classiﬁcation algorithms aims at effectively ﬁnding expressive representations of the original time series and then maps them into the ﬁnal prediction.

Original: This last operation is often realized via a relatively simple mapping, e.g. a linear map or a shallow neural network.
Revised: This last operation is often realized via a relatively simple mapping, e.g. a linear map or a shallow neural network.

Original: Deep learning algorithms used for TSC include various types of deep convolutional neural networks which extract a hierarchical representation of the input time series by learning the parameters of their ﬁlters via backpropgation.
Revised: Deep learning algorithms used for TSC include various types of deep convolutional neural networks which extract a hierarchical representation of the input time series by learning the parameters of their ﬁlters via backpropgation.

Original: These features are ﬁnally mapped into the logits associated with each class via a trainable linear projection.
Revised: These features are ﬁnally mapped into the logits associated with each class via a trainable linear projection.

Original: Last but not least, it is key to remark that these networks are typically over-parametrized and their training can be computationally demanding as well as challenging from an engineering perspective.
Revised: Last but not least, it is key to remark that these networks are typically over-parametrized and their training can be computationally demanding as well as challenging from an engineering perspective.

Original: In this section, we investigate how we can actually replace deep neural networks used for feature extraction with relatively cheap random layers which iteratively generate a reservoir.
Revised: In this section, we investigate how we can actually replace deep neural networks used for feature extraction with relatively cheap random layers which iteratively generate a reservoir.

Original: Among other notable works, Rocket (Dempster et al., 2020) arguably proposes the most successful solution.
Revised: Among other notable works, Rocket (Dempster et al., 2020) arguably proposes the most successful solution.

Original: Rocket.
Revised: Rocket.

Original: This algorithm prescribes the generation of k (cid:29) 1 unidimensional ﬁlters characterized by randomly drawn lengths, weights, and jumps.
Revised: This algorithm prescribes the generation of k (cid:29) 1 unidimensional ﬁlters characterized by randomly drawn lengths, weights, and jumps.

Original: Given a ﬁlter of length l filter , weights ω ∈ R l filter , and jump j ,  the feature Z i resulting from its action on an input path X , from position i in X , evolves according to the following iterative update:
Revised: Given a ﬁlter of length l filter , weights ω ∈ R l filter , and jump j , it can be easily shown that the feature Z i resulting from its action on an input path X , from position i in X , evolves according to the following iterative update:

Original: [Equation]
Revised: [Equation]

Original: where Z i 

Revised: where Z i

Original: =
Revised: =

Original: b and b is a random real number.
Revised: b and b is a random real number.

Original: We deﬁne Z i to be equal to Z il filter − 1 .
Revised: We deﬁne Z i to be equal to Z il filter − 1 .

Original: Depending on the values of j and l filter and on the length l X of the path X , this update produces a vector
Revised: Depending on the values of j and l filter and on the length l X of the path X , this update produces a vector

Original: Z =
Revised: Z =

Original: ( Z 1 , · · · , Z M ) of M :=
Revised: ( Z 1 , · · · , Z M ) of M :=

Original: l X − ( l − 1) · j random features, used to extract only two nonlinear features: max ( Z ) and 1 M (cid:80) M i =1 1 { Z i > 0 } .
Revised: l X − ( l − 1) · j random features, used to extract only two nonlinear features: max ( Z ) and 1 M (cid:80) Mi =1 1 { Z i > 0 } .

Original: Ultimately, this means that if we use k kernels, we extract 2 k features per time series which are then used as input to a Ridge classiﬁer to obtain the ﬁnal prediction.
Revised: Ultimately, this means that if we use k kernels, we extract 2 k features per time series which are then used as input to a Ridge classiﬁer to obtain the ﬁnal prediction.

Original: Rocket results in state-of-the-art performances in TSC both in terms of accuracy and computational efﬁciency, outperforming several deep learning baselines which often involve order of magnitude more parameters to optimize.
Revised: Rocket results in state-of-the-art performances in TSC both in terms of accuracy and computational efﬁciency, outperforming several deep learning baselines which often involve order of magnitude more parameters to optimize.

Original: An alternative interpretation.
Revised: An alternative interpretation.

Original: Looking at Eq.
Revised: Looking at Eq.

Original: 7, it is natural to wonder if it can be replaced with an alternative and possibly more principled update scheme driven by the input path and evolved by random projections.
Revised: 6, it is natural to wonder if it can be replaced with an alternative and possibly more principled update scheme driven by the input path and evolved by random projections.

Original: With the intent of aligning Rocket to the framework presented in this paper, we replace Eq.
Revised: With the intent of aligning Rocket to the framework presented in this paper, we replace Eq.

Original: 7 with the update equation in Algorithm 1, thus exploiting the expressive power of Randomized Signatures.
Revised: 6 with the update equation in Algorithm 1, thus exploiting the expressive power of Randomized Signatures.

Original: As detailed in the Appendix B.2, the current implementation of this algorithm is equivalent to assuming that the random vector ﬁeld A is approximately diagonal and autonomous.
Revised: As detailed in the Appendix B.2, the current implementation of this algorithm is equivalent to assuming that the random vector ﬁeld A is approximately diagonal and autonomous.

Original: In other words, the Randomized Signatures are never mixed during their evolution and, differently from the update in Eq.
Revised: In other words, the Randomized Signatures are never mixed during their evolution and, differently from the update in Eq.

Original: 7, the ﬁlter weights do not depend on time.
Revised: 6, the ﬁlter weights do not depend on time.

Original: These represent two strong limitations which pave the way to several enhancements in future work.
Revised: These represent two strong limitations which pave the way to several enhancements in future work.

Original: Despite these drastic shortcomings, the resulting algorithm outperforms several deep learning baselines on
Revised: Despite these drastic shortcomings, the resulting algorithm outperforms several deep learning baselines on

Original: TSC datasets from the UCR archive, as shown in Fig.
Revised: TSC datasets from the UCR archive, as shown in Fig.

Original: The original algorithm using Eq.provides better results than ours but does not enjoy the same theoretical guarantees based on the theory of Randomized Signatures presented in Section 3.
Revised: The original algorithm using Eq.provides better results than ours but does not enjoy the same theoretical guarantees based on the theory of Randomized Signatures presented in Section 3.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Reservoir Computing and the Concept of Signature.
Revised: Reservoir Computing and the Concept of Signature.

Original: The concept of reservoir computing is based on an simple and intriguing idea: generating a large number of features and then training a linear readout to perform an arbitrary downstream task.
Revised: The concept of reservoir computing is based on an simple and intriguing idea: generating a large number of features and then training a linear readout to perform an arbitrary downstream task.

Original: This simple approach promises to reduce the number of model parameters yet retaining a high level of expressive power.
Revised: This simple approach promises to reduce the number of model parameters yet retaining a high level of expressive power.

Original: Inspired by results from rough path theory showing that the Signature Transform enjoys the status of reservoir, we propose a random reservoir, i e. the Randomized Signature, to approximate the Signature Transform and thus inheriting its geometrical properties and expressive power.
Revised: Inspired by results from rough path theory showing that the Signature Transform enjoys the status of reservoir, we propose a random reservoir, i.e. the Randomized Signature, to approximate the Signature Transform and thus inheriting its geometrical properties and expressive power.

Original: Following rigorous theoretical arguments, we do so by evolving a latent differential equation controlled by a path and evolved according to random vector ﬁelds.
Revised: Following rigorous theoretical arguments, we do so by evolving a latent differential equation controlled by a path and evolved according to random vector ﬁelds.

Original: The empirical investigations presented in this paper provide evidence supporting that this strategy is successful and can be deployed on several tasks including non-parametric system identiﬁcation and TSC
Revised: The empirical investigations presented in this paper provide evidence supporting that this strategy is successful and can be deployed on several tasks including non-parametric system identiﬁcation and TSC.

Original: Future research and open questions.
Revised: Future research and open questions.

Original: From a theoretical standpoint, Eq. 4 is written in terms of autonomous vector ﬁelds
Revised: From a theoretical standpoint, Eq. 4 is written in terms of autonomous vector ﬁelds

Original: A i for i = 1 , · · · , d , whose entries are drawn according to Gaussian distributions with zero mean and unit variance, and an activation function σ .
Revised: A i for i = 1 , · · · , d , whose entries are drawn according to Gaussian distributions with zero mean and unit variance, and an activation function σ .

Original: It is natural to wonder if choosing time-dependent matrices
Revised: It is natural to wonder if choosing time-dependent matrices

Original: A i leads to increased expressive power and to what extent the quality of the resulting features depend on a particular choice of σ .
Revised: A i leads to increased expressive power and to what extent the quality of the resulting features depend on a particular choice of σ .

Original: Furthermore, since our experiments suggest that the dimensionality k of the random features play a signiﬁcant role in terms of expressive power, it is not trivial to identify a rigorous criterion to determine it.
Revised: Furthermore, since our experiments suggest that the dimensionality k of the random features play a signiﬁcant role in terms of expressive power, it is not trivial to identify a rigorous criterion to determine it.

Original: A precise characterization of this quantity and a clever way to select its value could potentially lead to further savings in terms of computational cost.
Revised: A precise characterization of this quantity and a clever way to select its value could potentially lead to further savings in terms of computational cost.

Original: From a practical point of view, our method proved to be able to accurately predict the response of different dynamical systems driven by heterogeneous and possibly irregular control paths, even when such controls were sampled from different laws with respect to the one used in training.
Revised: From a practical point of view, our method proved to be able to accurately predict the response of different dynamical systems driven by heterogeneous and possibly irregular control paths, even when such controls were sampled from different laws with respect to the one used in training.

Original: In light of these results, it would be interesting to explore whether the method of Randomized Signatures could be extended to model the response of systems governed by controlled partial differential equations .
Revised: In light of these results, it would be interesting to explore whether the method of Randomized Signatures could be extended to model the response of systems governed by controlled partial differential equations .

Original: In the context of TSC, we established a link with existing works using time series representations based on random features and our approach based on the update in Eq. 4.
Revised: In the context of TSC, we established a link with existing works using time series representations based on random features and our approach based on the update in Eq. 4.

Original: We implemented a ﬁrst baseline algorithm combining our Randomized Signatures with the algorithm proposed in (Dempster et al., 2020), resulting in superior performance compared to strong deep learning baselines but sub-optimal results compared to the original algorithm.
Revised: We implemented a ﬁrst baseline algorithm combining our Randomized Signatures with the algorithm proposed in (Dempster et al., 2020), resulting in superior performance compared to strong deep learning baselines but sub-optimal results compared to the original algorithm.

Original: We believe such a gap can be ﬁlled by addressing the limitation of the current version of the algorithm, e.g. the assumptions it makes on the vector ﬁelds, as discussed at the end of Section 5.
Revised: We believe such a gap can be ﬁlled by addressing the limitation of the current version of the algorithm, e.g. the assumptions it makes on the vector ﬁelds, as discussed at the end of Section 5.

Original: As a ﬁnal consideration, we believe an interesting research direction being to investigate whether our random feature extraction pipeline could be incorporated into hierarchical models, such as deep neural networks, to form hybrid methods including both random and learned components operating in symbiosis.
Revised: As a ﬁnal consideration, we believe an interesting research direction being to investigate whether our random feature extraction pipeline could be incorporated into hierarchical models, such as deep neural networks, to form hybrid methods including both random and learned components operating in symbiosis.

Original: Appendix A T HEORY D ETAILS A. B ASIC D EFINITIONS
Revised: Appendix A T HEORY D ETAILS A. B ASIC D EFINITIONS

Original: First of all, we deﬁne the concept of admissible tensor norms, which we assume to have in this work.
Revised: First of all, we deﬁne the concept of admissible tensor norms, which we assume to have in this work.

Original: Deﬁnition 2 (Admissible Tensor Norms)
Revised: Deﬁnition 2 (Admissible Tensor Norms)

Original: Let E := R d and ⊗ be a tensor product such that the tensor powers of E , (cid:0) E ⊗ k : k ≥ 1 (cid:1) , are equipped with a family ( (cid:107) · (cid:107) E ⊗ k : k ≥ 1) of norms satisfying:
Revised: Let E := R d and ⊗ be a tensor product such that the tensor powers of E , (cid:0) E ⊗ k : k ≥ 1 (cid:1) , are equipped with a family ( (cid:107) · (cid:107) E ⊗ k : k ≥ 1) of norms satisfying:

Original: For j, k ∈ N and for all h ∈ E ⊗ j and l ∈ E ⊗ k (cid:107) h ⊗ l (cid:107) E ⊗ ( j + k ) ≤ (cid:107) h (cid:107) E ⊗ j (cid:107) l (cid:107) E ⊗ k ; 2.
Revised: For j, k ∈ N and for all h ∈ E ⊗ j and l ∈ E ⊗ k (cid:107) h ⊗ l (cid:107) E ⊗ ( j + k ) ≤ (cid:107) h (cid:107) E ⊗ j (cid:107) l (cid:107) E ⊗ k ; 2.

Original: For any permutation σ of { 1 , . . . , k } , (cid:107) l 1 ⊗ . . .
Revised: For any permutation σ of { 1 , . . . , k } , (cid:107) l 1 ⊗ . . .

Original: ⊗ l k (cid:107) E ⊗ k
Revised: ⊗ l k (cid:107) E ⊗ k

Original: =
Revised: =

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: l σ (1) ⊗ . . .
Revised: l σ (1) ⊗ . . .

Original: ⊗ l σ ( k ) (cid:13)(cid:13)
Revised: ⊗ l σ ( k ) (cid:13)(cid:13)

Original: E ⊗ k ; 3.
Revised: E ⊗ k ; 3.

Original: For any bounded linear functionals f on E ⊗ j and g on E ⊗ k , there exists a unique bounded linear functional, denoted as f ⊗ g , on E ⊗ ( j + k ) such that for all h ∈ V ⊗ j and l ∈ E ⊗ k f ⊗ g ( h ⊗ l ) =
Revised: For any bounded linear functionals f on E ⊗ j and g on E ⊗ k , there exists a unique bounded linear functional, denoted as f ⊗ g , on E ⊗ ( j + k ) such that for all h ∈ V ⊗ j and l ∈ E ⊗ k f ⊗ g ( h ⊗ l ) =

Original: f ( h ) g
Revised: f ( h ) g

Original: ( l ) .
Revised: ( l ) .

Original: A family of tensor norms satisfying these conditions is called a family of admissible tensor norms.
Revised: A family of tensor norms satisfying these conditions is called a family of admissible tensor norms.

Original: Then we deﬁne the space in which the Signature of a path lies in, that is the following Tensor
Revised: Then we deﬁne the space in which the Signature of a path lies in, that is the following Tensor

Original: Algebra.
Revised: Algebra.

Original: Deﬁnition 3 (Tensor Algebra T (cid:0) R d (cid:1) )
Revised: Deﬁnition 3 (Tensor Algebra T (cid:0) R d (cid:1) )

Original: We deﬁne the tensor algebra on R d as
Revised: We deﬁne the tensor algebra on R d as

Original: [Equation]
Revised: [Equation]

Original: as well as its truncated version of order M ≥ 0 as
Revised: as well as its truncated version of order M ≥ 0 as

Original: [Equation]
Revised: [Equation]

Original: where (cid:0) R d (cid:1) ⊗ k is the space of tensors of shape ( d, . . .
Revised: where (cid:0) R d (cid:1) ⊗ k is the space of tensors of shape ( d, . . .

Original: , d ) given by R d ⊗ · · · ⊗ R d for k times.
Revised: , d ) given by R d ⊗ · · · ⊗ R d for k times.

Original: Deﬁnition 4 (Concatenation Operation)
Revised: Deﬁnition 4 (Concatenation Operation)

Original: We deﬁne the Concatenation Operation ∗ such that for any given couple of continuous piecewise smooth paths X : [0 , s ] → R d and Y : [ s, t ] → R d , their image through ∗ is the continuous piecewise smooth path X ∗ Y : [0 , t ] → R d deﬁned by
Revised: We deﬁne the Concatenation Operation ∗ such that for any given couple of continuous piecewise smooth paths X : [0 , s ] → R d and Y : [ s, t ] → R d , their image through ∗ is the continuous piecewise smooth path X ∗ Y : [0 , t ] → R d deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 5 (Inverse Operation)
Revised: Deﬁnition 5 (Inverse Operation)

Original: We deﬁne the Inverse Operation ←− such that for any continuous piecewise smooth path X : t → R d , its image through ←− is the continuous piecewise smooth path ←−
Revised: We deﬁne the Inverse Operation ←− such that for any continuous piecewise smooth path X : t → R d , its image through ←− is the continuous piecewise smooth path ←−

Original: X t := X T − t , for each t ∈ [0 , T ] .
Revised: X t := X T − t , for each t ∈ [0 , T ] .

Original: Deﬁnition 6 (Topological Space)
Revised: Deﬁnition 6 (Topological Space)

Original: A topological space is an ordered pair ( X, τ ) , where X is a set and τ is a collection of subsets of X , satisfying the following axioms:
Revised: A topological space is an ordered pair ( X, τ ) , where X is a set and τ is a collection of subsets of X , satisfying the following axioms:

Original: The empty set and X itself belong to τ ;
Revised: The empty set and X itself belong to τ ;

Original: Any arbitrary (ﬁnite or inﬁnite) union of members of τ still belongs to τ ; 3.
Revised: Any arbitrary (ﬁnite or inﬁnite) union of members of τ still belongs to τ ; 3.

Original: The intersection of any ﬁnite number of members of τ still belongs to τ .
Revised: The intersection of any ﬁnite number of members of τ still belongs to τ .

Original: The elements of τ are called open sets and the collection τ is called a topology on X .
Revised: The elements of τ are called open sets and the collection τ is called a topology on X .

Original: Deﬁnition 7 (Arcwise Connected Topological Space)
Revised: Deﬁnition 7 (Arcwise Connected Topological Space)

Original: A topological space ( X, τ ) is said to be arcwise connected if any two distinct points x, y ∈ X can be joined by an arc, that is a continuous map α : [0 , 1] → X such that α (0) =
Revised: A topological space ( X, τ ) is said to be arcwise connected if any two distinct points x, y ∈ X can be joined by an arc, that is a continuous map α : [0 , 1] → X such that α (0) =

Original: x and α (1) =
Revised: x and α (1) =

Original: y .
Revised: y .

Original: We say that X is uniquely arcwise connected if for any two distinct points x,
Revised: We say that X is uniquely arcwise connected if for any two distinct points x,

Original: y ∈ X , there exists a unique path in X that joins them.
Revised: y ∈ X , there exists a unique path in X that joins them.

Original: Deﬁnition 8 ( R -tree)
Revised: Deﬁnition 8 ( R -tree)

Original: An R -tree is a uniquely arcwise connected metric space, in which the arc between two points is isometric to an interval.
Revised: An R -tree is a uniquely arcwise connected metric space, in which the arc between two points is isometric to an interval.

Original: Deﬁnition 9 (Tree-Like)
Revised: Deﬁnition 9 (Tree-Like)

Original: A continuous piecewise smooth path x : [0 , T ] → R d is tree-like if there exists an R -tree τ , a continuous map φ : [0 , T ] → τ and a map ψ : τ → V such that φ (0) =
Revised: A continuous piecewise smooth path x : [0 , T ] → R d is tree-like if there exists an R -tree τ , a continuous map φ : [0 , T ] → τ and a map ψ : τ → V such that φ (0) =

Original: φ ( T ) and x = ψ ◦ φ .
Revised: φ ( T ) and x = ψ ◦ φ .

Original: A. S UPPORTING R ESULTS
Revised: A. S UPPORTING R ESULTS

Original: Lemma 4 (Johnson-Lindenstrauss Lemma, (Cuchiero et al., 2021a))
Revised: Lemma 4 (Johnson-Lindenstrauss Lemma, (Cuchiero et al., 2021a))

Original: Given an M -dimensional
Revised: Given an M -dimensional

Original: Hilbert space ( H, (cid:104)· , ·(cid:105) H ) and Q a N -point subset of H , for any 0 < (cid:15) < 1 , for each k ∈ N satisfying the so called Johnson-Lindenstrauss constraint
Revised: Hilbert space ( H, (cid:104)· , ·(cid:105) H ) and Q a N -point subset of H , for any 0 < (cid:15) < 1 , for each k ∈ N satisfying the so called Johnson-Lindenstrauss constraint

Original: [Equation]
Revised: [Equation]

Original: there exists a linear map f { Q,M,k } : H → R k that embeds Q into R k in an almost isometric manner.
Revised: there exists a linear map f { Q,M,k } : H → R k that embeds Q into R k in an almost isometric manner.

Original: More speciﬁcally, we have that
Revised: More speciﬁcally, we have that

Original: [Equation]
Revised: [Equation]

Original: f { Q

Original: [Equation]

Original: M [Equation] k } ( a 1 ) − f { Q [Equation]
Revised: f { Q,M,k } ( a 1 ) − f { Q,M,k } ( a 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (1 + (cid:15) )

Original: M [Equation] k } ( a 2 ) (cid:13)(cid:13)(cid:13) 2 ≤ (1 + (cid:15) ) (cid:107) a 1 − a 2 (cid:107) 2 H , for each a 1 , a 2 ∈ Q .
Revised: (cid:107) a 1 − a 2 (cid:107) 2 H , for each a 1 , a 2 ∈ Q .

Original: Remark 2 When there is no need to specify the dependence of f { Q,M,k } from Q , M , or k , we will omit them up to referring to is simply as f .
Revised: Remark 1 When there is no need to specify the dependence of f { Q,M,k } from Q , M , or k , we will omit them up to referring to is simply as f .

Original: Finally, we deﬁne f ∗ :
Revised: Finally, we deﬁne f ∗ :

Original: R k
Revised: R k

Original: → H to be the adjoint map of f with respect to a ﬁxed inner product (cid:104)· , ·(cid:105) in R k .
Revised: → H to be the adjoint map of f with respect to a ﬁxed inner product (cid:104)· , ·(cid:105) in R k .

Original: To apply such Lemma in our context, we select M ≥ 0 , equip T M (cid:0) R d (cid:1) with an inner product such that
Revised: To apply such Lemma in our context, we select M ≥ 0 , equip T M (cid:0) R d (cid:1) with an inner product such that

Original: [Equation]
Revised: [Equation]

Original: where { e i 1 ⊗ · · · ⊗ e i M } i 1 ,...,i M ∈{ 1 ,...,d } is the canonical basis of T M (cid:0) R d (cid:1) .
Revised: where { e i 1 ⊗ · · · ⊗ e i M } i 1 ,...,i M ∈{ 1 ,...,d } is the canonical basis of T M (cid:0) R d (cid:1) .

Original: Therefore, we have that (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) is an Hilbert space.
Revised: Therefore, we have that (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) is an Hilbert space.

Original: Theorem 5 (Existence and Uniqueness of the Signature, Lemma 2.10 (Terry J. Lyons, 2004))
Revised: Theorem 5 (Existence and Uniqueness of the Signature, Lemma 2.10 (Terry J. Lyons, 2004))

Original: The following controlled differential equation
Revised: The following controlled differential equation

Original: [Equation]
Revised: [Equation]

Original: has a unique solution which, at each t ∈ [0 , T ] is the Signature S t of X on [0 , t ] .
Revised: has a unique solution which, at each t ∈ [0 , T ] is the Signature S t of X on [0 , t ] .

Original: Notation 1 When there is no ambiguity about X , we will refer to its Signature as S , while we use
Revised: Notation 1 When there is no ambiguity about X , we will refer to its Signature as S , while we use

Original: S X if more paths are involved.
Revised: S X if more paths are involved.

Original: Theorem 6 (Theorem 2.29, (Terry J. Lyons, 2004))
Revised: Theorem 6 (Theorem 2.29, (Terry J. Lyons, 2004))

Original: Given a continuous piecewise smooth path
Revised: Given a continuous piecewise smooth path

Original: X : [0 , T ] → R d , its Signature is the banal Signature 1 :=
Revised: X : [0 , T ] → R d , its Signature is the banal Signature 1 :=

Original: (1 , 0 , 0 , · · · ) if and only if X is tree-like.
Revised: (1 , 0 , 0 , · · · ) if and only if X is tree-like.

Original: In particular S X ∗←− X = 1 .
Revised: In particular S X ∗←− X = 1 .

Original: Now, we provide results to show the relevance of these features.
Revised: Now, we provide results to show the relevance of these features.

Original: First of all, the following theorem ensures that the Signature of a path encodes its essence and characterizes it completely.
Revised: First of all, the following theorem ensures that the Signature of a path encodes its essence and characterizes it completely.

Original: According to Theorem 6, the concatenation X ∗ ←− X of X with its inverse 1 ←− X has the same Signature as the constant path, but cannot be reparametrised to be constant.
Revised: According to Theorem 6, the concatenation X ∗ ←− X of X with its inverse 1 ←− X has the same Signature as the constant path, but cannot be reparametrised to be constant.

Original: Similarly, if X, Y, Z are nonconstant paths, then S X ∗ Y ∗←− Y ∗ Z ∗←−
Revised: Similarly, if X, Y, Z are nonconstant paths, then S X ∗ Y ∗←− Y ∗ Z ∗←−

Original: Z ∗←− X = 1 , but X ∗ Y ∗ ←− Y ∗ Z ∗ ←− Z ∗ ←− X is not a path of the form γ ∗ ←− γ for any path γ .
Revised: Z ∗←− X = 1 , but X ∗ Y ∗ ←− Y ∗ Z ∗ ←− Z ∗ ←− X is not a path of the form γ ∗ ←− γ for any path γ .

Original: The formal deﬁnition of tree-like path is given in Deﬁnition 10.
Revised: The formal deﬁnition of tree-like path is given in Deﬁnition 10.

Original: Theorem 7 (Characterizing Nature of the Signature, Corollary 1.4 (Ben Hambly, 2010))
Revised: Theorem 7 (Characterizing Nature of the Signature, Corollary 1.4 (Ben Hambly, 2010))

Original: Given a couple of continuous piecewise smooth paths X and ˆ X , then S X = S
Revised: Given a couple of continuous piecewise smooth paths X and ˆ X , then S X = S

Original: ˆ X
Revised: ˆ X

Original: if and only if
Revised: if and only if

Original: X ∗ ←−
Revised: X ∗ ←−

Original: ˆ X is tree-like.
Revised: ˆ X is tree-like.

Original: This result is actually much stronger as it implies that the solution of any differential equation controlled by X is fully determined by the vector ﬁelds and S .
Revised: This result is actually much stronger as it implies that the solution of any differential equation controlled by X is fully determined by the vector ﬁelds and S .

Original: In particular, Theorem 1 shows that the solution of any differential equation controlled by X is essentially linear in S .
Revised: In particular, Theorem 1 shows that the solution of any differential equation controlled by X is essentially linear in S .

Original: Adapting Theorem III.8.
Revised: Adapting Theorem III.8.

Original: (Cuchiero et al., 2021b) as done in (Cuchiero et al., 2021a)
Revised: (Cuchiero et al., 2021b) as done in (Cuchiero et al., 2021a)

Original: one obtains:
Revised: one obtains:

Original: Theorem 8 (Randomized Signature)
Revised: Theorem 8 (Randomized Signature)

Original: For any ﬁxed integer M ≥ 0 , any ﬁxed partition D = { t 0 , · · · , t N } of t such that 0 ≤ t 0 < · · · < t N ≤ T , let us consider the Truncated Signature of order M of X at such times, that is Q := (cid:8) S Mt 0 , · · · , S Mt N (cid:9) such that its elements all lie in (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) .
Revised: For any ﬁxed integer M ≥ 0 , any ﬁxed partition D = { t 0 , · · · , t N } of t such that 0 ≤ t 0 < · · · < t N ≤ T , let us consider the Truncated Signature of order M of X at such times, that is Q := (cid:8) S Mt 0 , · · · , S Mt N (cid:9) such that its elements all lie in (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) .

Original: Let us now select 0 < (cid:15) < 1 , k ∈ N satisfying the associated JohnsonLindenstrauss constraint, let f be the implied Johnson-Lindenstrauss map and f ∗ its adjoint map.
Revised: Let us now select 0 < (cid:15) < 1 , k ∈ N satisfying the associated JohnsonLindenstrauss constraint, let f be the implied Johnson-Lindenstrauss map and f ∗ its adjoint map.

Original: Then, the solution of the controlled differential equation in R k
Revised: Then, the solution of the controlled differential equation in R k

Original: [Equation]
Revised: [Equation]

Original: on D , that is { Z t 0 , · · · , Z t N } , are called the Randomized Signature of X on D .
Revised: on D , that is { Z t 0 , · · · , Z t N } , are called the Randomized Signature of X on D .

Original: Each Z t k is the projection of S Mt k from (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) onto R k via f and thus preserves its geometric properties and its approximation power.
Revised: It holds that each Z t k is the projection of S Mt k from (cid:0) T M (cid:0) R d (cid:1) , (cid:104)· , ·(cid:105) (cid:1) onto R k via f . To conclude, since f ∗ ( Z t k ) is close to S Mt k in the norm of T M (cid:0) R d (cid:1) , we conclude that Z preserves the geometric properties and the approximation power of S .

Original: Adapting Theorem III.7 in (Cuchiero et al., 2021b)
Revised: Adapting Theorem III.7 in (Cuchiero et al., 2021b)

Original: it can be be shown that (asymptotically) the JL projected vector ﬁelds stem from random matrices:
Revised: it can be be shown that (asymptotically) the JL projected vector ﬁelds stem from random matrices:

Original: Theorem 9 ( Z is a random dynamical system) For k (cid:55)−→ ∞ , for each i ∈ { 1 , · · · , d } the linear vector ﬁelds f { k } (cid:0) f { k }∗ ( · ) e i (cid:1) : R k
Revised: Theorem 9 ( Z is a random dynamical system) For k (cid:55)−→ ∞ , for each i ∈ { 1 , · · · , d } the linear vector ﬁelds f { k } (cid:0) f { k }∗ ( · ) e i (cid:1) : R k

Original: → R k are square matrices with asymptotically normally distributed, independent entries.
Revised: → R k are square matrices with asymptotically normally distributed, independent entries.

Original: Leveraging this results, we deﬁne the following:
Revised: Leveraging this results, we deﬁne the following:

Original: Deﬁnition 10 (Localized Randomized Signature)
Revised: Deﬁnition 10 (Localized Randomized Signature)

Original: For any random matrices A 1 , . . .
Revised: For any random matrices A 1 , . . .

Original: , A d in R k × k and shifts b 1 , . . .
Revised: , A d in R k × k and shifts b 1 , . . .

Original: , b d in R k × 1 such that maximal non-integrability holds on a random starting point z ∈ R k , any ﬁxed activation functionfunction σ , and d -dimensional control X , the solution of
Revised: , b d in R k × 1 such that maximal non-integrability holds on a random starting point z ∈ R k , any ﬁxed activation function σ , and d -dimensional control X , the solution of

Original: [Equation]
Revised: [Equation]

Original:  is called the Localized Randomized Signature of X and we often refer to it as Randomized Signature.
Revised: 7) is called the Localized Randomized Signature of X and we often refer to it as Randomized Signature.

Original: Theorem 10 (Signature is a Reservoir (Restatement of Theorem 1)) Let V i : R m → R m , i = 1 , . . .
Revised: Theorem 10 (Signature is a Reservoir (Restatement of Theorem 1)) Let V i : R m → R m , i = 1 , . . .

Original: , d be vector ﬁelds regular enough such that dY t = (cid:80) d i =1 V i
Revised: , d be vector ﬁelds regular enough such that dY t = (cid:80) d i =1 V i

Original: ( Y t )
Revised: ( Y t )

Original: dX i t , Y 0 =
Revised: dX i t , Y 0 =

Original: y ∈ R m , admits a unique solution Y t : [0 , T ] → R m .
Revised: y ∈ R m , admits a unique solution Y t : [0 , T ] → R m .

Original: Then, for any smooth test function F : R m → R and for every
Revised: Then, for any smooth test function F : R m → R and for every

Original: M ≥ 0 there is a time-homogeneous linear operator L : T M (cid:0) R d (cid:1)
Revised: M ≥ 0 there is a time-homogeneous linear operator L : T M (cid:0) R d (cid:1)

Original: → R which depends only on ( V 1 , . .
Revised: → R which depends only on ( V 1 , . .

Original: , V d , F, M, y ) such that F ( Y t ) =
Revised: , V d , F, M, y ) such that F ( Y t ) =

Original: L (cid:0) S Mt (cid:1) + O (cid:0) t M +1 (cid:1) , and t ∈ [0 , T ] .
Revised: L (cid:0) S Mt (cid:1) + O (cid:0) t M +1 (cid:1) , and t ∈ [0 , T ] .

Original: [Equation]
Revised: [Equation]

Original: Proof: We provide just a sketch.
Revised: Proof: We provide just a sketch.

Original: For every smooth test function F : R m → R , the formal Taylor expansion along controls together with the fundamental theorem of calculus gives us that
Revised: For every smooth test function F : R m → R , the formal Taylor expansion along controls together with the fundamental theorem of calculus gives us that

Original: [Equation]
Revised: [Equation]

Original: For y ∈ R m , this equation can be inserted into itself leading to a generalized version of Taylor expansion for controlled ordinary differential equations
Revised: For y ∈ R m , this equation can be inserted into itself leading to a generalized version of Taylor expansion for controlled ordinary differential equations

Original: [Equation]
Revised: [Equation]

Original: for M ≥ 0 , with the remainder satisfying
Revised: for M ≥ 0 , with the remainder satisfying

Original: [Equation]
Revised: [Equation]

Revised: A. P RACTICAL EXAMPLE OF THE COMPUTATION OF S IGNATURES ( ONE DIMENSION ).

Revised: To enhance intuition, we exemplify the computation of Signatures in the simplest setting possible.

Revised: Let X : [0 , T ] → R , then S 1 t = (cid:82) t 0 dX s — which is exactly X t − X 0 .

Revised: To get S 2 t , we instead have to compute the following iterated integral:

Revised: S 2 t = (cid:82) t 0 (cid:0)(cid:82) v 0 dX s (cid:1) dX v .

Revised: Signatures of higher order S jt are computed in a similar way, by iteratively integrating the path j times.

Revised: As a practical example, let X t = t .

Revised: Then it is easy to see that S jt = t j j ! .

Revised: Now let Y ( t ) be an analytic function of time for which we have Y ( t )

Revised: = (cid:80) ∞

Revised: j =0 Y ( j ) 0 t j j ! .

Revised: Taylor’s theorem combined with the previous computation of the Signatures implies that Y can be approximated as a linear map of the Signatures of t .

Revised: Finally, note that S jt in this case gets smaller and smaller in magnitude as j increases.

Revised: This suggests that the truncated Signature can be safely used to approximate Y .

Revised: Computational Complexity and dimensionality of Signatures.

Revised: The computational complexity for computing S Mt is O

Revised: ( d M ) .

Revised: Indeed, consider d = 2 : S 2 t is a 2 × 2 matrix with elements (cid:82) t 0 (cid:0)(cid:82) v 0 dX 1 s (cid:1) dX 1 v , (cid:82) t 0 (cid:0)(cid:82) v 0 dX 1 s (cid:1) dX 2 v , (cid:82) t 0 (cid:0)(cid:82) v 0 dX 2 s (cid:1) dX 1 v

Revised: and (cid:82) t 0 (cid:0)(cid:82) v 0 dX 2 s (cid:1) dX 2 v .

Revised: For M = 3 , the object to compute is instead a 2 × 2 × 2 tensor, containing all integrals of the type (cid:82) t 0 (cid:0)(cid:82) w 0 (cid:0)(cid:82) v 0 dX i 1 s (cid:1) dX i 2 v (cid:1) dX i 3 w for all i 1 , i 2 , i 3 ∈ { 1 , 2 } .

Revised: Hence, the complexity — as well as the features dimensionality — scales exponentially in M .

Original: B E XPERIMENTS
Revised: B E XPERIMENTS

Original: In this section, we prove that our approach can accurately learn the dynamics of several rough differential equations.
Revised: In this section, we prove that our approach can accurately learn the dynamics of several rough differential equations.

Original: First of all, we focus on dynamics of 1 -dimensional stochastic
Revised: First of all, we focus on dynamics of 1 -dimensional stochastic

Original: Double
Revised: Double

Original: Well which is characterized by a highly nonlinear dynamics.
Revised: Well which is characterized by a highly nonlinear dynamics.

Original: Secondly, we study the multidimensional case of the 4 -dimensional Ornstein–Uhlenbeck.
Revised: Secondly, we study the multidimensional case of the 4 -dimensional Ornstein–Uhlenbeck.

Original: Then, we investigate the performance on the rough dynamics of Fractional Geometric Brownian Motion even for choices of extremely low values of the hurst parameter H .
Revised: Then, we investigate the performance on the rough dynamics of Fractional Geometric Brownian Motion even for choices of extremely low values of the hurst parameter H .

Original: To conclude, we obtain promising results in learning the dynamics of an Enzyme-Substrate Reactions.
Revised: We also obtain promising results in learning the dynamics of an EnzymeSubstrate Reactions.

Revised: Next, we show how in high dimensions randomized signatures are computationally efﬁcient, and also present one experiment in comparison with neural controlled differential equations (Kidger et al., 2020).

Revised: To conclude, we show how the randomized signature approach can be used in cases where the time grid is irregularly sampled.

Original: B. P RELIMINARY R ESULTS : R OUGH D IFFERENTIAL E QUATIONS
Revised: B. P RELIMINARY R ESULTS : R OUGH D IFFERENTIAL E QUATIONS

Original: 1 -Dimensional Stochastic Double
Revised: 1 -Dimensional Stochastic Double

Original: Well.
Revised: Well.

Original: Let us recall that the dynamics of the 1 -Dimensional
Revised: Let us recall that the dynamics of the 1 -Dimensional

Original: Stochastic Double Well process is given by
Revised: Stochastic Double Well process is given by

Original: [Equation]
Revised: [Equation]

Original: where W t is a 1 -dimensional Brownian motion, and ( µ, θ, σ ) ∈ R × R + × R + .
Revised: where W t is a 1 -dimensional Brownian motion, and ( µ, θ, σ ) ∈ R × R + × R + .

Original: Let us ﬁx y 

Revised: Let us ﬁx y

Original: = 1 and ( µ = 2 , θ = 1 , σ = 1) , and the partition D of [0 , 1] to have N = 101 equally spaced time steps.
Revised: = 1 and ( µ = 2 , θ = 1 , σ = 1) , and the partition D of [0 , 1] to have N = 101 equally spaced time steps.

Original: We train a Ridge Regression with regularization parameter λ = 0 .
Revised: We train a Ridge Regression with regularization parameter λ = 0 .

Original: 001 to map instances of Randomized
Revised: 001 to map instances of Randomized

Original: Signature of the controls
Revised: Signature of the controls

Original: Z into the respective solution Y t .
Revised: Z into the respective solution Y t .

Original: We repeat the experiment on different values of the number N Train of train samples and dimension k of the Randomized Signature.
Revised: We repeat the experiment on different values of the number N Train of train samples and dimension k of the Randomized Signature.

Original: On the left of Figure 8, we plot an example of the trajectory of Z while, on its right, we plot the comparison of the true and the generated time series on an out-of-sample case.
Revised: On the left of Figure 9, we plot an example of the trajectory of Z while, on its right, we plot the comparison of the true and the generated time series on an out-of-sample case.

Original: The following table shows the performance in terms of L 2 relative error on 10000 test samples:
Revised: The following table shows the performance in terms of L 2 relative error on 10000 test samples:

Original: N Train =
Revised: N Train =

Original: N Train = N Train =
Revised: N Train = N Train =

Revised: [Figure]

Original: N Train =
Revised: N Train =

Revised: [Figure]

Original: N Train =
Revised: N Train =

Revised: [Figure]

Original: k =0. k =0.
Revised: k =0. k =0.

Original: k
Revised: k

Original: =0.
Revised: =0.

Revised: [Figure]

Original: 4 -Dimensional Ornstein–Uhlenbeck process.
Revised: 4 -Dimensional Ornstein–Uhlenbeck process.

Original: Let us recall that the dynamics of the 4Dimensional Ornstein–Uhlenbeck process is given by
Revised: Let us recall that the dynamics of the 4Dimensional Ornstein–Uhlenbeck process is given by

Original: [Equation]
Revised: [Equation]

Original: 1.0 Time 7.

Original: V a l u e

Original: Random Signatures (8/111 plotted) 0.

Original: 1.0 Time 0.

Original: V a l u e Out Of Sample Extracted Y t True Y t where W t is a 4 -dimensional Brownian motion, and ( µ , Θ , Σ ) ∈ R 4 × R 4 × 4 × R 4 × 4 .
Revised: where W t is a 4 -dimensional Brownian motion, and ( µ , Θ , Σ ) ∈ R 4 × R 4 × 4 × R 4 × 4 .

Original: Let us ﬁx y 

Revised: Let us ﬁx y

Original: = 1 , µ = 1 ,
Revised: = 1 , µ = 1 ,

Original: Σ = 1 4 , Θ i.j = i/j , the partition D of [0 , 1] to have N = 101 equally spaced time steps, and k = 708 .
Revised: Σ = 1 4 , Θ i.j = i/j , the partition D of [0 , 1] to have N = 101 equally spaced time steps, and k = 708 .

Original: Finally, we train a Ridge Regression with λ = 0 .
Revised: Finally, we train a Ridge Regression with λ = 0 .

Original: 001 on N Train train sample and Figure 9 shows the comparison of out-of-sample generated and true trajectories while Tablereports the performance in terms of L 2 relative error on 10000 test samples.
Revised: 001 on N Train train sample and Figure 10 shows the comparison of out-of-sample generated and true trajectories while Tablereports the performance in terms of L 2 relative error on 10000 test samples.

Original: k =
Revised: k =

Original: NTrain =
Revised: N

Original: N Train = N Train = N Train =
Revised: [Table]

Original: N Train = 
Revised: = [Table] N

Original: Y0. Y0. Y0. Y0.
Revised: [Table]

Original: 1 -Dimensional Geometric Fractional Brownian Motion.
Revised: 1 -Dimensional Geometric Fractional Brownian Motion.

Original: Let us recall that the dynamics of the 1 -Dimensional Geometric Fractional Brownian Motion process given by
Revised: Let us recall that the dynamics of the 1 -Dimensional Geometric Fractional Brownian Motion process given by

Original: [Equation]
Revised: [Equation]

Original: where µ ∈ R , σ ∈ R \ { 0 } , and B ( H ) t is a one-dimensional fractional Brownian motion of hurst parameter H ∈ (0 , 1) .
Revised: where µ ∈ R , σ ∈ R \ { 0 } , and B ( H ) t is a one-dimensional fractional Brownian motion of hurst parameter H ∈ (0 , 1) .

Original: In this experiment, we ﬁx y 

Revised: In this experiment, we ﬁx y

Original: = 1 and ( µ = 1 , σ = 1)
Revised: = 1 and ( µ = 1 , σ = 1)

Original: , we let H vary in { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } , and the partition D of [0 , 1] is made of N equally spaced times.
Revised: , we let H vary in { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } , and the partition D of [0 , 1] is made of N equally spaced times.

Original: We highlight that, as per Theorem D.3.2 in (Biagini et al., 2008), when H < 1 / 4 , the fractional Brownian motion is so rough that it is no longer a Rough Path and its Signature is no longer deﬁned.
Revised: We highlight that, as per Theorem D.3.2 in (Biagini et al., 2008), when H < 1 / 4 , the fractional Brownian motion is so rough that it is no longer a Rough Path and its Signature is no longer deﬁned.

Original: Finally, we train a Ridge Regression with λ = 0 .
Revised: Finally, we train a Ridge Regression with λ = 0 .

Original: 001 on N Train =
Revised: 001 on N Train =

Original: 10000 train samples of Z into the logarithm of the respective solution log( Y t ) which implies that we need to then apply the exp function to our predictions.
Revised: 10000 train samples of Z into the logarithm of the respective solution log( Y t ) which implies that we need to then apply the exp function to our predictions.

Original: The right of Figure 10 shows the comparison of the generated and true trajectories on an out-of-sample trajectory while the following table shows the Relative L 2 Error as we vary the number of time steps N and hurst parameter H :
Revised: The right of Figure 11 shows the comparison of the generated and true trajectories on an out-of-sample trajectory while the following table shows the Relative L 2 Error as we vary the number of time steps N and hurst parameter H :

Original: H = 0 . H = 0 . H = 0 . H = 0 .
Revised: [Equation]

Original: H = 0 . N =2 . 44 · 
Revised: N =2 . 44 · 10 −7 .

Original: [Figure] [Figure]
Revised: 07 · 10 −5 .

Original: [Figure] [Figure] [Table] [Figure]
Revised: 73 · 10 −

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: −

Original: [Table] [Figure]
Revised: [Table] [Table]

Original: = [Figure] [Figure] [Figure]
Revised: −

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: N =3 .

Original: [Table] [Figure] [Table] [Figure]
Revised: 04 · 10 −

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: −

Original: [Table] [Figure]
Revised: [Table] [Table]

Original: =
Revised: −

Original: [Figure] [Table]
Revised: [Table] [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: N =

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: Enzyme-Substrate Reactions.
Revised: Enzyme-Substrate Reactions.

Original: First, we consider a set of highly nonlinear CODEs modeling the interaction between a catalyst (enzyme, with abundance C t ) and a substrate S (with abundance S t ).
Revised: First, we consider a set of highly nonlinear CODEs modeling the interaction between a catalyst (enzyme, with abundance C t ) and a substrate S (with abundance S t ).

Original: Substrate (e.g. lactose) is injected via the control variable X t , reacts with the enzyme (e.g. lactase) and the concentration Y t of the product chemical (e.g. glucose) is observed (Ingalls, 2013).
Revised: Substrate (e.g. lactose) is injected via the control variable X t , reacts with the enzyme (e.g. lactase) 1.0 Time 0. V a l u e

Original: [Equation]
Revised: Out Of Sample Extracted Y t True Y t 0. 1.0 Time 0.

Original:  dS
Revised: V a l u e Out Of Sample Extracted Y t True Y t and the concentration Y t of the product chemical (e.g. glucose) is observed (Ingalls, 2013).

Original: [Equation]
Revised: [Equation]

Original: Following (Ingalls, 2013) we choose ( k 1 , k 2 ) =
Revised: Following (Ingalls, 2013) we choose ( k 1 , k 2 ) =

Original: (30 , 10) , set ( S 0 , P 0 , Y 0 )
Revised: (30 , 10) , set ( S 0 , P 0 , Y 0 )

Original: = (0 , 0 , 0) and consider the evolution on 0 ≤ t ≤ 1 .
Revised: = (0 , 0 , 0) and consider the evolution on 0 ≤ t ≤ 1 .

Original: We ﬁx the time grid to have N = 101 equally spaced time steps and the control X t to follow the law of W 2 t where W t is a 1 -dimensional Brownian Motion (to ensure positivity).
Revised: We ﬁx the time grid to have N = 101 equally spaced time steps and the control X t to follow the law of W 2 t where W t is a 1 -dimensional Brownian Motion (to ensure positivity).

Original: We train a Ridge Regression with regularization parameter λ
Revised: We train a Ridge Regression with regularization parameter λ

Original: = 10 − 3 to map 10instances of 222 -dimensional Randomized Signature Z of the controls into the respective solution
Revised: = 10 − 3 to map 10instances of 222 -dimensional Randomized Signature Z of the controls into the respective solution

Original: Y t .
Revised: Y t .

Original: On the left of Figure 11, we plot the comparison of the true and the generated time series on a test sample.
Revised: On the left of Figure 12, we plot the comparison of the true and the generated time series on a test sample.

Original: As we can see, the model has learnt to correctly map a trajectory of X t to the respective system response Y t .
Revised: As we can see, the model has learnt to correctly map a trajectory of X t to the respective system response Y t .

Original: More surprisingly, the right of such a ﬁgure shows that our model is able to predict the correct output even if we stimulate the system with substrate injection that follows a completely different law such as 0 .
Revised: More surprisingly, the right of such a ﬁgure shows that our model is able to predict the correct output even if we stimulate the system with substrate injection that follows a completely different law such as 0 .

Original: 5 · 1 { W 2 t >
Revised: 5 · 1 { W 2 t >

Original: [Figure]
Revised: [Figure]

Original: This suggests that the system was correctly identiﬁed out of distribution.
Revised: This suggests that the system was correctly identiﬁed out of distribution.

Original: [Figure]
Revised: [Figure]

Revised: Comparison with Neural Controlled Differential Equations In this experiment, we benchmark the performance of our Randomized Signature model with that of Neural Controlled Differential

Revised: Equations (NCDEs) (Kidger et al., 2020), a recently proposed competitive deep learning technique for controlled dynamical system modelling.

Revised: We test the two models on the task of learning the dynamics of the 1 -dimensional Ornstein-Uhlenbeck Process.

Revised: As usual, to train our model, we ﬁx an equally spaced partition D of [0 , 1] , extract the Randomized Signature of the controls for N Train samples, and train a Ridge Regression to map them into the respective integrated solutions.

Revised: The

Revised: NCDE models instead parametrize the potentials of a latent controlled differential equation with feed-forward neural networks with 1 hidden layer of n nodes .

Revised: Figure 13 shows the comparison of our model against several version of the NCDEs.

Revised: Additionally, Table 4 and Table 5 show the number of trainable parameters, average L 2 Relative Errors, and training times for all the models for N Train = 100 and N Train = 1000 respectively.

Revised: We observe that our model is the best one from all aspects and that not even a considerable amount of training data is beneﬁcial for NCDEs.

Revised: [Table]

Revised: [Table]

Original: B. A DDITIONAL D ETAILS
Revised: A DDITIONAL D ETAILS

Original: Learning the Signatures.
Revised: Learning the Signatures.

Original: In this experiment, we ﬁx the partition D
Revised: In this experiment, we ﬁx the partition D

Original: = { t 0 , · · · , t N } of [0 , 1] to have 201 equally-spaced time steps and the regularization parameter of the Ridge Regression is λ = 0 .
Revised: = { t 0 , · · · , t N } of [0 , 1] to have 201 equally-spaced time steps and the regularization parameter of the Ridge Regression is λ = 0 .

Original: Reconstruction and Randomized Signature Autoencoders.
Revised: Reconstruction and Randomized Signature Autoencoders.

Original: In this experiment, we ﬁx the partition D
Revised: In this experiment, we ﬁx the partition D

Original: = { t 0 , · · · , t N } of [0 , 1] to have 101 equally-spaced time steps, ﬁx k = 111 , and consider a single realization X Train of an Ornstein–Uhlenbeck process of asymptotic mean µ = 2 , mean reversion θ = 1 , and volatility σ = 1 .
Revised: = { t 0 , · · · , t N } of [0 , 1] to have 101 equally-spaced time steps, ﬁx k = 111 , and consider a single realization X Train of an Ornstein–Uhlenbeck process of asymptotic mean µ = 2 , mean reversion θ = 1 , and volatility σ = 1 .

Original: We train a Ridge Regression with regularization parameter λ = 0 .
Revised: We train a Ridge Regression with regularization parameter λ = 0 .

Original: 001 to map the Randomized Signature Z of the one and only train sample X Train into
Revised: 001 to map the Randomized Signature Z of the one and only train sample X Train into

Original: X Train itself.
Revised: X Train itself.

Revised: k =

Revised: k =k =

Revised: N Test =

Original: [Table]
Revised: [Table]

Revised: Comparison with Vanilla Signature Approach Let us recall that the dynamics of the dDimensional Fractional Ornstein–Uhlenbeck process is given by

Revised: [Equation]

Revised: where B Ht is a d -dimensional Fractional Brownian motion of hurst parameter H ∈ (0 , 1) , and ( µ , Θ , Σ ) ∈ R d × R d × d × R d × d .

Original: {+Let us ﬁx y 
+}
Revised: {+Let us ﬁx y 
+}

Revised: = 1 , µ = 1 , Σ = 1 d , Θ i.j = i/j , the partition D of [0 , 1] to have N = 101 equally spaced time steps, and H = 0 .

Original: 1 -Dimensional Stochastic Double Well - Irregularly Sampled Time Grid.
Revised: 1 -Dimensional Stochastic Double Well - Irregularly Sampled Time Grid.

Original: [-In this experiment, we ﬁx y 
-]
Revised: [-In this experiment, we ﬁx y 
-]Let us recall that the dynamics of the 1 -Dimensional Stochastic Double Well process is given by [Equation] where W t is a 1 -dimensional Brownian motion, and ( µ, θ, σ ) ∈ R × R + × R + . For each train and test sample, the partition D of [0 , 1] is made of N randomly drawn times.

Revised: More precisely, D = { 0 , t 1 , · · · , t N − 1 , 1 } such that t k = 1 / (1 − exp( − s k )) and { s 1 , · · · , s N − 1 } are N − 2 independent realizations of a uniform distribution U [0 , 1] sorted increasingly.

Revised: As a result, the probability that two samples share the same D is null.

Revised: We train a Ridge Regressor with N Train = 10000 train samples and

Revised: Figure 11 shows the comparison of an out-of-sample generated and true trajectory.

Revised: Finally, Table 7 shows the Relative L 2 Error on 10000 test samples as we vary the number of time steps N and k and while comparing it to the respective experiment in case the time grid is regularly spaced.

Revised: As we can see, even though the performance are worse than the regularly sampled setup, this technique proves to be anyway reliable on irregularly sampled regimes.

Original: {+In this experiment, we ﬁx y 
+}
Revised: {+In this experiment, we ﬁx y 
+}

Original: = 1 and ( µ = 2 , θ = 1 , σ = 1) while the regularization parameter of the Ridge Regression is λ = 0 . 
Revised: = 1 and ( µ = 2 , θ = 1 , σ = 1) while the regularization parameter of the Ridge Regression is λ = 0 . [Equation] [Equation]

Revised: Irregular 0 .0 .

Revised: [Equation]

Revised: Regular 0 .0 .

Original:  Tumor Growth Kinetics.
Revised: [Equation] Tumor Growth Kinetics.

Original: We set ( k 1 , k 2 , λ 0 , λ 1 , Ψ) =
Revised: We set ( k 1 , k 2 , λ 0 , λ 1 , Ψ) =

Original: (10 , 0 . 5 , 0 . 9 , 0 . 7 , 20) , and (cid:0) u 10 , u 20 , u 30 , u 40 , Y 0 (cid:1) =
Revised: (10 , 0 . 5 , 0 . 9 , 0 . 7 , 20) , and (cid:0) u 10 , u 20 , u 30 , u 40 , Y 0 (cid:1) =

Original: We ﬁx the partition D of [0 , 10] to have 101 equally spaced time steps and we again train a Ridge Regression with regularization parameter λ
Revised: We ﬁx the partition D of [0 , 10] to have 101 equally spaced time steps and we again train a Ridge Regression with regularization parameter λ

Original: = 0 .to map 10000 instances of the 222 -dimensional Randomized Signature Z of the controls into the respective solution Y t .
Revised: = 0 .to map 10000 instances of the 222 -dimensional Randomized Signature Z of the controls into the respective solution Y t .

Original: Electrochemical Battery Model.
Revised: Electrochemical Battery Model.

Original: In this experiment, we ﬁx the partition D of [0 , 500] to have 1001 equally spaced time steps, the regularization parameter of the Ridge Regression is λ = 10 −while k ∈ { 50 , 166 , 1000 } .
Revised: In this experiment, we ﬁx the partition D of [0 , 500] to have 1001 equally spaced time steps, the regularization parameter of the Ridge Regression is λ = 10 −while k ∈ { 50 , 166 , 1000 } .

Original: Regarding the latest experiment, we simply learn to map the Random
Revised: Regarding the latest experiment, we simply learn to map the Random

Original: Signature of the controls into the respective Y to which we added white noise with variance 0 .
Revised: Signature of the controls into the respective Y to which we added white noise with variance 0 .

Original: Regarding NNARX, for the regressor we follow the results presented in (Masti and Bemporad, 2018),
Revised: Regarding NNARX, for the regressor we follow the results presented in (Masti and Bemporad, 2018),

Original: where n a = n b = 12 , and n k = 1 lead to the best results.
Revised: where n a = n b = 12 , and n k = 1 lead to the best results.

Original: Regarding the neural network, we use a feedforward neural network with input dimension n a + n b + 1 = 25 and 2 hidden layers each with either 22 (NNARX22) or 1000 (NNARX1000) hidden units.
Revised: Regarding the neural network, we use a feedforward neural network with input dimension n a + n b + 1 = 25 and 2 hidden layers each with either 22 (NNARX22) or 1000 (NNARX1000) hidden units.

Original: For the loss function, we used the mean square error optimized using Adam with learning rate 0 .
Revised: For the loss function, we used the mean square error optimized using Adam with learning rate 0 .

Original: 01 for 100 epochs.
Revised: 01 for 100 epochs.

Original: Table 6 reports the performances in terms of MSE on 1000 test samples for our models compared to NNARX.[Table]
Revised: Table 8 reports the performances in terms of MSE on 1000 test samples for our models compared to NNARX.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: We run our TSC experiments on the UCR archive, one of the most popular benchmarks for TSC methods (Dau et al., 2019; Fawaz et al., 2019).

Original: [Table]
Revised: It consists of 128 heterogeneous datasets of different sizes comprising possibly irregularly sampled one-dimensional time series of various lengths. Rocket is one of the best performing algorithms on this benchmark.

Original: Rocket.
Revised: R OCKET

Original: When implementing Rocket, we generate k kernels each characterized by a certain random length l filter , random weights ω ∈ R l filter , and random jump j .
Revised: When implementing Rocket, we generate k kernels each characterized by a certain random length l filter , random weights ω ∈ R l filter , and random jump j .

Original: Since the lengths and the jumps are randomly sampled, the length M of the vector Z =
Revised: Since the lengths and the jumps are randomly sampled, the length M of the vector Z =

Original: ( Z 1 , · · · , Z M ) of the random features varies accordingly.
Revised: ( Z 1 , · · · , Z M ) of the random features varies accordingly.

Original: Given that the number of possible combinations of the values of l filter and j is limited, we can group the kernels based on such combinations.
Revised: Given that the number of possible combinations of the values of l filter and j is limited, we can group the kernels based on such combinations.

Original: As a result, among each of these groups, the length M is ﬁxed and, additionally, each kernel is applied to the very same sub-timeseries (cid:0) X i ,
Revised: As a result, among each of these groups, the length M is ﬁxed and, additionally, each kernel is applied to the very same sub-timeseries (cid:0) X i ,

Original: X i +j , · · · , X i +( l filter − 1) × j (cid:1) of X starting from position i .
Revised: X i +j , · · · , X i +( l filter − 1) × j (cid:1) of X starting from position i .

Original: If we now focus on one of these subgroups and let us assume it contains ˜ k kernels of length l filter and jump j .
Revised: If we now focus on one of these subgroups and let us assume it contains ˜ k kernels of length l filter and jump j .

Original: As a result of the application of each kernel on X , we obtain the following matrix  
Revised: As a result of the application of each kernel on X , we obtain the following matrix  

Original: Z 0 , Z 1 ,. . . Z M,
Revised: [Equation] [Equation]

Original: Z 0 , Z 1 ,. . . Z M,...
Revised: [Equation] [Equation]

Original: Z 0 , ˜ k Z 1 , ˜ k . . .
Revised: [Equation]

Original: Z M, ˜ k   where Z i,r is the result of the action of the r -th kernel on the sub-time-series (cid:0)
Revised: k   where Z i,r is the result of the action of the r -th kernel on the sub-time-series (cid:0)

Original: X i , X i +j , · · · , X i +( l filter − 1) × j (cid:1) of X starting from position i .
Revised: X i , X i +j , · · · , X i +( l filter − 1) × j (cid:1) of X starting from position i .

Original: Finally, out of each row, we extract the features described above.
Revised: Finally, out of each row, we extract the features described above.

Original: Turning to our approach, we generate k kernels each characterized by a certain length l filter , weights ω
Revised: Turning to our approach, we generate k kernels each characterized by a certain length l filter , weights ω

Original: =
Revised: =

Original: ( ω 0 , ω 1 ) ∈ R 2 , and jump j .
Revised: ( ω 0 , ω 1 ) ∈ R 2 , and jump j .

Original: As before, let us again group them based on all the possible combinations of l filter and j and focus on only one of such subgroups.
Revised: As before, let us again group them based on all the possible combinations of l filter and j and focus on only one of such subgroups.

Original: As a consequence, we have ˜ k kernels each characterized by weights ( ω r 0 , ω r 1 ) , r ∈ { 1 , · · · , ˜ k } with ﬁxed length and jump.
Revised: As a consequence, we have ˜ k kernels each characterized by weights ( ω r 0 , ω r 1 ) , r ∈ { 1 , · · · , ˜ k } with ﬁxed length and jump.

Original: The application of each of these ˜ k kernels following the update rules proposed in Algorithm 1 is equivalent to extracting a 1 -dimensional Randomized Feature of each of the sub-time-series (cid:0) 
Revised: The application of each of these ˜ k kernels following the update rules proposed in Algorithm 1 is equivalent to extracting a 1 -dimensional Randomized Feature of each of the sub-time-series (cid:0) X i , X i + [Equation]

Original: X i , X i +j , · · · , X i +( l filter − 1) × j (cid:1) .
Revised: [Equation] , X [Equation] +( l filter − 1) × j (cid:1) .

Original: As a result, we obtain the following matrix  
Revised: As a result, we obtain the following matrix  

Original: Z 0 ,
Revised: Z 0 ,

Original: Z 1 ,. . .
Revised: Z 1 ,. . .

Original: Z M,
Revised: Z M,

Original: Z 0 ,
Revised: Z 0 ,

Original: Z 1 ,. . .
Revised: Z 1 ,. . .

Original: Z M,...
Revised: Z M,...

Original: Z 0 , ˜ k
Revised: Z 0 , ˜ k

Original: Z 1 , ˜ k . . .
Revised: Z 1 , ˜ k . . .

Original: Z M, ˜ k   where Z i,r is the 1 -dimensional Randomized Signature of (cid:0) X 0 , X j , · · · , X ( l filter − 1) × j (cid:1) via the r -th kernel.
Revised: Z M, ˜ k   where Z i,r is the 1 -dimensional Randomized Signature of (cid:0) X 0 , X j , · · · , X ( l filter − 1) × j (cid:1) via the r -th kernel.

Original: Let us we deﬁne the random matrix
Revised: Let us we deﬁne the random matrix

Original: A guiding the evolution of the Randomized Signature as the ˜ k ×
Revised: A guiding the evolution of the Randomized Signature as the ˜ k ×

Original: ˜ k diagonal matrix having the weights ω r 0 on the diagonal while the random bias b is a column vector having the ω r 1 as entries.
Revised: ˜ k diagonal matrix having the weights ω r 0 on the diagonal while the random bias b is a column vector having the ω r 1 as entries.

Original: Then, the i -th column is the ˜ k -dimensional Randomized Signature of the sub-time-series (cid:0) X
Revised: Then, the i -th column is the ˜ k -dimensional Randomized Signature of the sub-time-series (cid:0) X

Original: i , X i +j , · · · , X i +( l filter − 1) × j (cid:1) .
Revised: i , X i +j , · · · , X i +( l filter − 1) × j (cid:1) .

Original: Finally, out of each row, we extract the features described above.
Revised: Finally, out of each row, we extract the features described above.

Original: Comparison of Computational Complexities and Dimensions
Revised: Comparison of Computational Complexities and Dimensions

Original: The following table reports key ﬁgures regarding the computational complexity and dimensionality of both Signature and Randomized Signature.
Revised: The following table reports key ﬁgures regarding the computational complexity and dimensionality of both Signature and Randomized Signature.

Original: Computational complexity
Revised: Comp. complexity

Original: Dimension
Revised: Dimension

Original: Guarantees
Revised: Guarantees

Original: Truncated Sign.
Revised: Truncated Sign.

Original: ( M - components):
Revised: ( M - components):

Original: [Equation]
Revised: O ( d M ) O ( d M ) (Thm 1)

Original: Randomized Sign.
Revised: Randomized Sign.

Original: ( d - dimensional):
Revised: ( d - dimensional):

Original: [Equation]
Revised: O ( k 2 d ) O ( k ) (Thm 2)