Paper: A Note on Some Statistical Properties of Signature Transform Under Stochastic Integrals (ICLR-2024)

Reviewer comment (Writing):
I have thought that the sentence “Given the successful application … time series data” expresses the motivation for this study. However, this motivation is too abstract. I would like to see a concrete description as to why discussion to support claim 1 is demanded.

Author response:
We appreciate the reviewer's feedback and we agree wholeheartedly with the reviewer that establishing the motivation for our research is important. In the revised manuscript, we have modified the paragraph to better convey the motivation behind our work.

We believe that discussion to support Claim 1 is necessary for three key reasons. First, to the best of our knowledge, most empirical literature on signatures uses signatures directly without considering their underlying statistical properties. Second, the probabilistic foundation of empirical research using signatures, the universal nonlinearity, has been explored under different definitions of signatures. However, the associated statistical properties for different definitions of signatures have not been examined. Third, to our knowledge, most empirical research uses a default definition of signatures regardless of the specific scenarios under investigation. However, using an inappropriate signature definition may lead to suboptimal performance, as highlighted by our work. Together, these three reasons motivate us to study the statistical performance of signatures.

Reviewer comment (Experiments):
In recent years, “series-to-point non-linear (neural-network-based) regression”, which is an end-to-end regression in the setting of this paper, has been well studied. For example, studies referred in Section 3.3 of “Ahmed, S., Nielsen, I. E., Tripathi, A., Siddiqui, S., Ramachandran, R. P., & Rasool, G. (2023). Transformers in time-series analysis: A tutorial. Circuits, Systems, and Signal Processing, 42(12), 7433-7466”. Is there a practical advantage to doing “point-to-point linear regression via signature transformation”? (I may have missed a cited paper that makes such a comparison. I'm sorry if that's the case. Please tell me that paper.) Considering readers of ICLR, it is worth mentioning the comparison between such techniques and signature-based ones. Also, the input dimension $\frac{d^{K+1}-1}{d-1}$ of “point-to-point linear regression via signature transformation” can be larger than the input dimension $d T$ of “end-to-end regression”. Is this a negative factor of “point-to-point linear regression via signature transformation”?

Author response:
We appreciate the reviewer's insightful question and agree that comparing neural-network-based regression with the signature-based model can make our paper more appealing to readers of ICLR. 

**Practical advantages of signature transform.** We believe that the signature transform has two practical advantages. First, it enhances computational efficiency in signature-based models because it only requires training a linear model; second, the linear model allows for interpretability. Essentially, the signature transform can be understood as a powerful method of feature engineering with sound theoretical properties. We have incorporated this comparison into Section 1 of the revised manuscript and have provided an interpretation of signatures in Appendix A. 

**Comparison between input dimensions.** We acknowledge the reviewer's concern about potentially higher input dimensions for the signature-based model ($\frac{d^{K+1}-1}{d-1}$) compared to the neural-network-based model ($dT$). However, we do not think this is a negative factor for the signature-based model for two main reasons. 

First, signatures are computed using the original $dT$ input data. Therefore, $dT$ can also be regarded as the input dimension for the signature-based model. On the other hand, neural-network-based models typically involve a large number of parameters to be trained, significantly increasing their overall dimensions. In fact, as mentioned above, the signature transform itself is a feature engineering step that allows the subsequent step to use a simple linear model.

Second, $\frac{d^{K+1}-1}{d-1}$ is not a concern in practice because, it has been documented that including signatures up to a small order $K$ usually suffices to achieve good performances [7,8] (see Section 2.2 of our paper).

Reviewer comment (Writing):
I would like to ask for additional explanation of interpretations of signature $ S(\mathbf{X}\_n)\_T^{i\_1,\ldots,i\_k}$ for each combination of $i\_1,\ldots,i\_k$. There are generally two main types of evaluation strategies of Lasso: regression error or parameter estimate error. This research focuses on the latter, which means emphasizing the interpretability of the parameter estimate itself. For that position, the interpretability of signatures should be important.

Author response:
We thank the reviewer for the comment, and we agree wholeheartedly with the reviewer that it is important to derive intuition and interpretation from signatures. In the revised manuscript, we have added a new appendix, Appendix A, which introduces the geometric interpretation of signatures. This interpretation has been documented in previous works such as [9].

Reviewer comment (Experiments):
I could not understand the meaning of the experimental comparisons for Figures 1 and 2 to support claim 1. When integral methods are different, the generated data (in particular, $\\{y\_n\\}$) are different, and data analysis methods (the sets of features used for Lasso regression) are also different. For example, for data analysis in real situation, only one data is given, so analysis with different generated data is useless. In particular, since the comparison uses different data, I think that there is a gap between the experimental results and claim 1. Contrary, if the authors want to focus their discussion on the influence of the integral method on the distribution properties of the signature, it would be natural to consider with a single data analysis method. Please tell me more specifically what the authors want to claim with the experimental comparisons for Figures 1 and 2, and in what situations that claim is useful.

Author response:
We thank the reviewer's question. Our objective is to compare the model consistency achieved using different *data analysis methods* (the set of features used for Lasso regression). More precisely, we want to investigate whether the consistency of Lasso is higher when using Itô signatures or Stratonovich signatures as the feature set. 

To ensure that the consistency is well defined, it is crucial that the set of features included in the Lasso ($A^* \cup A^{\*c}$) contains all true features of the true model ($A^*$). Therefore, we have to assume that $y_n$ is generated based on either Itô or Stratonovich signatures depending on which type of signatures we are using as features. Hence, when using different integral methods, different data $y_n$ are generated, and different data analysis methods are used.

We understand that this setup might seem confusing when comparing Itô and Stratonovich signatures. However, because our goal is to compare different data analysis methods, we have to adopt this setup. For example, assume that the data $y_n$ is generated using Itô signatures. Then, given the data, it would be unfair to compare the consistency of Itô signatures with that of Stratonovich signatures, as the latter would be undefined (or, alternatively, we can say the latter one is zero, because no true signatures could be selected).

In our numerical studies presented in Figures 1 and 2, we compute the consistency rates for Itô (Stratonovich) signatures assuming that the data $y_n$ is generated by Itô (Stratonovich) signatures, and make a comparison between them. This is useful in the following sense. For example, assume that we are given a set of real data $\{(\mathbf{X}_n, y_n)\}$, where $\mathbf{X}_n$ is a Brownian motion. We do not know whether $y_n$ is generated by Itô or Stratonovich signatures of $\mathbf{X}_n$. From the Bayesian perspective, we have a noninformative prior that $y_n$ is generated by either Itô or Stratonovich signatures each with a probability of 50\%. Our numerical results in Figure 1 imply that using Itô signatures to analyze the data would yield a higher expected model consistency, because 
$$
0.5 \times \text{Itô signatures' consistency rate} + 0.5 \times 0 > 0.5 \times 0 + 0.5 \times \text{Stratonovich signatures' consistency rate}.    
$$

Reviewer comment (Evaluation):
Another problem is that consideration and explanation of reasons for the experimental results are not sufficiently described. (I imagine that multicollinearity among the explanatory variables $ S(\mathbf{X})\_T^{\ldots}$ influences the experimental results; is it right?)

Author response:
We thank the reviewer for the comment. Indeed, the reviewer is correct that the collinearity between $S(\mathbf{X})_T$ influences the results. This collinearity also depends on the integral's definition. In our revised manuscript, we have added additional explanations to our results in Section 4.1.

Reviewer comment (Theory):
Regarding “Other feature selection techniques” in Section 5. The ridge regression is not used for feature selection typically. Rather, it will be better to cite, for example, bridge regression (Frank, I. E., and Friedman, J. H. (1993), “A Statistical View of Some Chemometrics Regression Tools,” Technometrics, 35, 109–135).

Author response:
We appreciate the reviewer's feedback. We have revised the manuscript according to the reviewer's suggestion.

Reviewer comment (Theory):
$\\{\epsilon\_n\\}\_n$ in (3) requires 0-mean assumption at least (otherwise $\tilde{\beta}\_0$ will be biased).

Author response:
We appreciate the reviewer's comment. In the revised manuscript, we have clarified that $\varepsilon_n$ should have a zero mean and finite variance.

Reviewer comment (Novelty):
It is not entirely clear to me that Lasso regression combined with signature transforms constitutes a widespread methodology used by numerous researchers/practitioners. The references listed do not seem to be enough to support this. Furthermore, in the introduction it is claimed that, on several important Machine Learning problems, this methodology yields state-of-the-art results. But many of the references provided to support this are rather old for the standards of the field. Thus, it is not clear that this is indeed the case. All lead me to think that, although the paper has clear merits, it could maybe be a better fit for a more specialized venue than ICLR. However, if the authors could provide further references to change this opinion, I would be willing to upgrade my evaluation.

Author response:
We appreciate the reviewer's feedback. In the revised manuscript, we have included more related literature which provides evidence that the signature has been increasingly adopted in various tasks in machine learning. First, we have incorporated additional literature on the combination of Lasso with signature transforms in the paragraph "Consistency of Lasso" in Section 1. Second, we have included more recent papers when discussing the applications of signatures in various fields in the first paragraph of Section 1. Third, to illustrate the alignment of our paper with ICLR, we have included several signature-related references that are accepted by top computer science conferences, such as [1] at ICLR, [2] at NeurIPS, and [3] at CVPR. We hope that these additional discussions of related literature enhance the completeness of our manuscript and help address the reviewer's concerns.

Reviewer comment (Evaluation):
The second one being that consistency does not appear to be the most relevant performance metric for this kind of problems. First because it only describes the large sample limit of the estimator and does not give insights about its performance for, the usually more realistic, finite sample scenarios. But also because, as discussed by the authors themselves, this measure can be too restrictive. It is true that the authors go on to explore other performance metrics in the appendices; but they do so only in a numerical way. Finally, sign consistency cannot be defined in a misspecified setting.

Author response:
We thank the reviewer for this comment. We choose sign consistency as the starting point of our research because it is a basic and well-accepted metric for studying Lasso [4,5,6]. In addition, although our theoretical results are based on large sample limits, the finite sample numerical results presented in our paper align with these theoretical results, implying that the insights gained from the large sample analysis can be informative in finite sample scenarios as well. We also believe, with all due respect, it is fairly standard in the literature to use large sample asymptotics for theoretical foundations and verify its practical effectiveness in finite sample using simulation.

In the meantime, we completely agree with the reviewer that exploring theoretical results in finite sample settings and considering alternative metrics beyond consistency is valuable. Other metrics, such as the out-of-sample $R^2$, may also alleviate the reviewer's concern that sign consistency is undefined in a misspecified setting. Our paper is an initial exploration of the statistical properties related to signature selection using Lasso, which does not exist in the literature. Therefore we choose consistency as the starting point. We would like to conduct more in-depth research on the finite sample case and other metrics in our future work.

Reviewer comment (Theory):
Although the results presented rely on the Irrepresentable condition, little is discussed about it. It is also stated that it "is almost a necessary and sufficient condition for Lasso estimator to be sign consistent". But the meaning of "almost" is never explained. It is my opinion that, if some further discussion were added on this respect, the results could be better appreciated.

Author response:
We thank the reviewer for the feedback. In our revised manuscript, we have added more explanations regarding the irrepresentable condition and clarified the meaning of "almost" in Appendix C. In particular, we introduce different versions of sign consistency and irrepresentable condition, and explain their relationship. In fact, we borrow the word "almost" precisely from the classical reference of [4] in this literature.

Reviewer comment (Theory):
Is the bound in Proposition 7 expected to be tight under certain regime? When is it loose? It would be good if some details about this could be added.

Author response:
We appreciate the reviewer's insightful question. In Appendix E of our revised manuscript, we investigate a specific scenario where the inter-dimensional correlations of the Brownian motion, denoted as $\rho = \rho_{ij}$, are the same. In this setup, we prove that the bound given by Proposition 7 is tight when $\rho<0$, and is loose when $\rho>0$. This is a very helpful supplement to our manuscript, so we really appreciate the reviewer for raising this question.

Reviewer comment (Evaluation):
I think it would be good if some heuristic interpretation of why different performances for signatures defined with Ito and Stratonovich integrals are observed.

Author response:
We thank the reviewer's feedback, and we agree with the reviewer wholeheartedly that adding heuristic interpretations can improve our manuscript. In fact, the different performances of Itô and Stratonovich signatures can be intuitively attributed to the difference in the definitions of Itô and Stratonovich integrals. In our revised manuscript, we have added these explanations at the end of Section 4.1.