Paper: Interpretable Sparse System Identification: Beyond Recent Deep Learning Techniques on Time-Series Prediction (ICLR-2024)

Reviewer comment (Presentation):
Inline citations are not in brackets, which sometimes makes the text hard to follow.

Author response:
* **C.1 Inline citations are not in brackets:** We have added brackets for each citations to improve the clarity.

Reviewer comment (Writing):
Acronyms are often not defined.

Author response:
* **C.2 Acronyms are often not defined:** We have included the full names of relatively less familiar models such as TRMF (Temporal Regularized Matrix Factorization) and TLAE (Temporal Latent Auto-Encoder) in the manuscript when they first occured. For commonly used models like RNN, SVM, LSTM, GRU, etc., due to space constraints, we still keep using the abbreviations.

Reviewer comment (Writing):
There are some typos throughout the work.

Author response:
* **C.3 Some typos throughout the work:** We have thoroughly checked the entire manuscript to reduce typos and improve presentation. For example, modifying the formula (3) in section 3.1.1, rectifying instances in Chapter 3 where "training set" was erroneously written as "test set," and resolving language issues in section 4.1.4. Moreover, corrections have been made for some capitalization and spelling errors.

Reviewer comment (Evaluation):
Percentage improvement is only given for MSE. Additional analysis of percentage improvement for MAE would be helpful (potentially in the appendix).

Author response:
* **Q.1 Percentage improvement of MAE:** Thanks for suggestion. We have put them in Appendix D.3.

Reviewer comment (Experiments):
I have a small concern regarding data leakage from the test set. I believe the test batch information is only used for local prediction (constructing the storage basis), but in Figure 1 there is an arrow going from the test batch back into the global prediction (predicted length). Could the authors please verify that there is no data leakage that would affect the global prediction/overall results.

Author response:
* **Q.2 Concern regarding data leakage:** We apologize for the confusion caused by the incorrect use of arrows in Figure 1. The arrow associated with "Prediction Length" indicates that we input the length of the local prediction into the training set. Subsequently, we perform the DFT on the data in the training set with this specified length, forming a basis for storage. This involves only inputting the length value, for instance, predicting 96 steps would entail inputting the value 96, without including any values from the test set. Hence, there is no issue of data leakage. We have modified the arrows labeled "Prediction Length" to indicate directly from the dataloader since the data loader can provide the output length. We believe this reduces any further misunderstandings.

Reviewer comment (Experiments):
How are values for $\alpha$ and $\beta$ selected? The work explains what high/low values mean and when they should be chosen, but how were they determined in practice?

Author response:
* **1. Hyperparameters:** In Appendix D.4, we have provided a detailed explanation of the hyperparameter selection process. The hyperparameters $\alpha$ and $\beta$ are conditionally tuned. When the mean difference between global predictions and input exceeds the half range of the input data, indicating a significant global prediction error that may affect local predictions, we set $\alpha=0.8, \beta=0.9$. Otherwise, we set $\alpha=0.1, \beta=0.9$. There are two benefits to this approach to adjust $\alpha$ and $\beta$. On the one hand, $\alpha$ and $\beta$ are not very sensitive to prediction results, so they do not require too much fine-tuning, since conditional tuning process makes it efficient. On the other hand, it prevents the disruption of local predictions by bad global prediction results.

Reviewer comment (Reproducibility):
Section 4.1.3 gives a vague measure of "seconds to minutes" for running the experiments. Are the authors able to provide concrete training/inference times for GLIP, and compare this to the same for the baselines?

Author response:
* **2. Concrete times for GLIP:** Specific computation time for GLIP and neural networks is as follows. For instance, on the ETTh dataset, the computation time is as follows. The unit after the values is second.

| Prediction Length | GLIP | DLinear | Autoformer |
|:---------------:|:--:|:-:|:-:|
|      96      |  116s  |  189s  |  299s  |
|      192     |  122s  |  216s  |  325s  |
|      336     |  138s  |  239s  |  378s  |
|      720     |  139s  |  244s  |  412s  |

Here, GLIP operates on a CPU enviornment, whereas the other two neural network models were run with support of GPU, Nvidia RTX A6000. Even under such conditions, our processing speed remains superior to that of neural networks. This illustrates the advantage of GLIP from the prespective of running time.

Reviewer comment (Writing):
complex language makes the text more arduous to understand than necessary

Author response:
* **W.1 Language issues:** We have carefully checked the language issues in subsection 4.1.4. Additionally, we have rephrased the cotents in Chapter 3. To enhance clarity in the new version, we have added multiple citations referring to individual panels in Figure 1 in subsections 3.1.1, 3.1.2 and 3.3.2. Moreover, we have included pseudocode for the entire algorithm in Appendix. E to facilitate a better understanding for the readers.

Reviewer comment (Reproducibility):
no details on hyperparameter tuning provided in the appendix.

Author response:
* **W.2 Details on hyperparameter tuning:** We have relocated the hyperparameter tuning process to Appendix D. 4. For specific questions regarding certain hyperparameters, we have provided additional answers to Q.3 below.

Reviewer comment (Reproducibility):
what do the authors mean exactly with "reasonable" in their statement: code is available upon **reasonable** request? Will the source code be published on GitHub? If not, what is the motivation behind not providing the code open source?

Author response:
* **Q.1 Code:** Open-source code has been provided in the supplementary material. It is convenient for users to reproduce the experiments. Thus, we have deleted the sentence, "code is available upon reasonable request" in the new version.

Reviewer comment (Writing):
In Section 3.1.1, while the section describes a Discrete Fourier Transformation (DFT) on X, it is not clear where in this section the resulting A, f are used.

Author response:
**A**: Through the application of ***DFT***, we can transform time series into the frequency domain, where $\mathbf{A}$ represents the amplitude and $\mathbf{f}$ signifies the frequency. $\mathbf{A}$ and $\mathbf{f}$ are vectors with the equal length, maintaining a one-to-one correspondence. The larger the amplitude, the greater the impact of the corresponding frequency on the time series. Our objective is to select, based on the amplitude magnitude, the frequencies most likely to construct the main components of the time series. Subsequently, these selected frequencies are transformed into basis functions for consideration, which will be further screened through sparse identification. Therefore, our candidate periodical terms $\mathbf{T^*}=[T_1, T_2,...,T_{m_1}]$ are determined by the selection process involving $\mathbf{A}$ and $\mathbf{f}$.  We have incorporated this selection process into Appendix. C and referenced it in the main text after the statement "For further global prediction, frequencies with high amplitude are selected" in the new version.

Reviewer comment (Writing):
In Section 3.1.1, it is not clear what T/T* are or how they are computed. Is \Theta_g including exactly three items or multiple sin/cos waves? This question also applies to Section 3.1.2.

Author response:
**A**: After a series of potential frequencies $\mathbf{f}$ influencing the time series were selected based on the higher amplitudes $\mathbf{A}$, a series of potential periods $T_i$ were subsequently derived by employing the frequency-period conversion formula $f=1/T$. This forms a row vector $\mathbf{T^*}=[T_1, T_2,...,T_{m_1}]$, where each element records a potential period most likely to influence the given time series. Similarly, $\mathbf{C}_1$ is also a row vector, as mentioned in the Notation section regarding the computation of operations involving scalars and vectors. The construction of matrix $\mathbf{\Theta_g}$ is not limited to three basic items/functions. Taking $\sin(\mathbf{C}_1\mathbf{t})$ as an example, where $\mathbf{C}_1$ is an $m_1 \times 1$ row vector and $\mathbf{t}$ is a $1 \times n$ column vector, they form an $m_1 \times n$ matrix. Thus, $\sin(\mathbf{C}_1\mathbf{t})$ is also an $m_1 \times n$ matrix. The construction of $\sin(\mathbf{C}_2\mathbf{t})$ is similar, and the final $\mathbf{1}$ represents a column vector of all ones. Therefore, there should be a total $2m_1 +1$ basic functions. This explanation is equally applicable to sections 3.2 and 3.3. To enhance clarity, when $\mathbf{T^*}$ and $\mathbf{t}$ are first introduced in Section 3.1, we have added row and column vector labels for them in the new version.

Reviewer comment (Writing):
In Section 3.1.1, the phrase "from the univariate predictions with the test set" should probably be "training set".

Author response:
**A**: In the old version, indeed, "from the univariate predictions with the test set," should be "from the univariate predictions with the training set". We have corrected it in the new version.

Reviewer comment (Theory):
In Section 3.1.1, X^*_g is not really used in Eq. (3), which is different from what the text states. Also, it is not clear what order of expressions were considered.

Author response:
**A**: The functions formed by $\mathbf{X}$ consisting the expression of $\mathbf{\Theta}_g^*$ in Eq. (3) should be $\mathbf{X}^*$. We have corrected it in the new version. The order of expressions is 2. This will be explained in Appendix (D.4).

Reviewer comment (Presentation):
In Section 3.2, it is not clear whether this is part of the model/training or a description of how to validate the model's performance (e.g., the paper mentions it is used to "evaluate" or "conduct an assessment").

Author response:
**A**: In Section 3.2, our validation set is used to evaluate the effectiveness of the training set. The results from the validation set here will impact the subsequent use of local rolling predictions. We have provided pseudocode in Appendix. E, and its role can be inferred from line 14 and line 21 of the pseudocode. Our approach follows the neural network paradigm, dividing the data into training, validation, and test sets. Here, the role of the validation set is similar to that in neural network methods.  To enhance clarity in our expression, we have rephrased the description in subsection 3.2.1 and rewritten subsection  3.2.2. We hope that, such a revision could help improve the understanding of the role of the validation set in GLIP.

Reviewer comment (Reproducibility):
The paper would significantly benefit from the inclusion of a pseudo-code describing training and inference procedures.

Author response:
**A**: Thanks for the suggestion. We have given a pseudo-code in the Appendix. E to improve the presentation, and we referenced it after introducing the method in subsection 3.3.3.

Reviewer comment (Experiments):
One significant weakness is related to interpretability. The paper positions the proposed approach as an interpretable approach (including in the title and in the motivation described in the abstract). However, the paper presents no results on interpretability and only includes a short description stating that the parameters are "amenable to interpretation". To make strong claims regarding the interpretability of the proposed approach, some results are needed (i.e., what are these parameters, are they weights associated with interpretable features, how many are there, etc.). Some example(s) for the interpretability of results would be very useful as well.

Author response:
**A**: We suggest interpretability pervades GLIP and it can be explained from three aspects. Firstly, GLIP employs DFT to explicitly identify potential periods within the time series for both global and local identification, incorporating them into the basis functions. This can be utilized to interpret the main components extracted from the frequency domain of the time series. Secondly, in comparison with neural networks, GLIP has an explicit mathematical function expression for time series prediction tasks. The function expression provides insights into the predicted trends with an explicit white-box form. It is worth noting that in the article of N-BERTS model mentioned in the review, the "Interpretability" section in N-BERTS also discusses the explicit representation of trend model and seasonality model. Thirdly, the hyperparameters, namely $k_1, k_2, \alpha, \beta,\lambda$, and hyperparameters $\Xi$ have practical significance and can be used directly to interpret the model. Comparatively, hyperparameters such as the number of hidden layers and neurons in neural networks or the number of multi-heads in Transformer may not have straightforward interpretability. Overall, we believe that GLIP, at least compared to most neural network based prediction methods, possesses interpretability to a large degree. We have further elaborated on this point in subsection 4.1.4 in the new version.

Reviewer comment (Experiments):
In the experiments, regarding datasets, the number of datasets used in the paper seems significantly smaller compared to comparable works. For example, in [2] and [3], other datasets such as weather, electricity, and traffic were included.

Author response:
**A**: GLIP has one advantage that the computational capability is supported by CPU so as to be more suitable for datasets with lower variable dimensions, since we can use sufficient candidate functions to model the inter-coupling relationships between these variables. For datasets with a high dimension of variables, constructing finite basis functions becomes a big challenge in accurately capturing the underlying mechanistic features of the data. Additionally, as the dimension of variables increases, the composition scale of basis functions grows exponentially, leading to the curse of dimensionality in optimization computations. GPU does not support the algorithm for optimization process of system identification in datasets with a very high dimension of variables. That is, system identification algorithms do not rely on computational power requirements to some extent, which cannot benefit from the powerful computation support from GPU. Therefore, deep learning algorithms, supported by GPU, may demonstrate advantages for datasets with a very high dimension of variables. E.g., in the datasets with less than twenty dimensions of variables presented in this study, GLIP exhibits clear advantages. However, as the number of variables increases, modeling such time series with hundreds of thousands of basis functions (multipliers) becomes a huge challenge for accurately reconstructing and predicting the systems. We have mentioned this in the Conclusion and Future work section in the old version. However, this issue is not insurmountable. One solution is that we have tested GLIP as a pre-trained module, integrated into the architecture of neural network. This integration widely improves reconstruction and prediction effects. Yet, this integrated approach sacrifices the interpretability and low computational requirements inherent in standalone system identification algorithms. Therefore, this study does not delve into the detailed exploration of this integrated architecture. Another solution is that we randomly selected ten inter-coupling variables for the target variable in the multi-variable dataset with more than e.g., twenty variables. Such a slight implement change of GLIP also indicates advantages compared with Dlinear and Autoformer, see the following Table. Here, the bold numbers represent the best prediction results, and the italic numbers mean the second-best results. Additionally, with a better selction mechanism of inter-coupling variables, the performence of GLIP for large-scale multi-variable datasets can be further improved.

Reviewer comment (Novelty):
In the literature review, the work should probably mention other models that have focused on basis expansion, such as N-BEATS [1].

Author response:
**A**: Thanks for the suggestion. We have cited and accordingly added an explanation for interpretability in the fourth paragraph of the new Introduction section.

Reviewer comment (Writing):
Minor issue in Section 3: "in the testing set for global prediction" should be "training set".

Author response:
**A**: Revised with many thanks.