Paper: Learning to Generate Predictor for Long-Term Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
The proposed method includes multiple modules and each of them have their own hyper-parameters to tune. Extensive hyper-parameter tuning was not favorable.

Author response:
- Hyperparameter
    - For the hyperparameter sensitivity analysis, we added the results in appendix B6. Since our LGPred is a network that generates the network, the sensitivity to the hyperparameters related to the predictor generation is somewhat inevitable. However, we believe that our methodology of generating the predictor adaptively to the given time series has clear value even considering this limitation.

Reviewer comment (Theory):
I was not clear why a bottleneck architecture was used for template predictor and intuition behind this design was not discussed.

Author response:
- Bottleneck architecture for template predictor
    - The reason we utilize the bottleneck architecture for our template predictor is to reduce the number of parameters to generate. As we mentioned in page 5, the number of parameters in predictor generator is significantly reduced due to the bottleneck architecture, although the number is still large. Also, the intuition behind the bottleneck architecture is the fact that the rank required for the linear predictor to predict time series is low. We conduct an experiment lowering the rank of the linear predictor by dividing the linear predictor into a bottleneck architecture of two linear layers having small latent dimension and measuring the performance. Our experiment results show that the performance is preserved even though the rank is much lower than the input and output length (d_latent =32). While subsequent research revealed that a larger latent dimension (rank) is required for predictor generation in most cases, the effectiveness of the bottleneck architecture in linear predictors and the effect of reducing the complexity of the bottleneck architecture are still meaningful.

Reviewer comment (Writing):
Some parts in the Preliminary and Method sections are not clear. It is not clear why the number of channels is changed in the trend block.

Author response:
Reason why the number of channels in Trend block in Fig. 2
    - During feature extraction, we decided to maintain the length of time dimension in the feature, to preserve the temporal information. However, for the channel dimension, since our generated predictor is applied in channel independence manner, we chose to mix the channel information and encode it into the features with length of d_rep instead of preserving it. This is the reason why we change the number of channels in the trend block of Fig.2, as explained in page 4.

Reviewer comment (Writing):
Some parts in the Preliminary and Method sections are not clear. The dilated temporal convolutional network should be introduced, in case some readers do not have related background knowledge.

Author response:
Explanation about dilated temporal convolution network
    - Regarding the dilated temporal convolution network, we added some explanation about dilated TCN and why we utilize dilated TCN for seasonality representation module in page 4.

Reviewer comment (Experiments):
The comparison with baselines may be unfair and experiments are insufficient. It is better to add the results of PatchTST in Table 3 due to its superiority.

Author response:
Comparison with PatchTST
    - In Table 3, we did not include the result of PatchTST in our first submission. We obtained the scores of baseline models in Table 3 from the TimesNet paper [2], and the performance of PatchTST is not included in the original paper. Note that the results in short input length situations have not been reported in the PatchTST paper [3]. However, we ran additional experiments using the implementation in Time-Series-Library (https://github.com/thuml/Time-Series-Library), which is an official implementation of TimesNet, and added the results in Table 3. The result shows that LGPred outperforms PatchTST when the input length is limited in most cases.

Reviewer comment (Experiments):
The comparison with baselines may be unfair and experiments are insufficient. There is no complexity analysis between the proposal and baselines.

Author response:
Complexity analysis
    - For the complexity analysis, we added the results in appendix B.7. Thank you for your comment and please check section B.7 in appendix of our updated paper.

Reviewer comment (Experiments):
The comparison with baselines may be unfair and experiments are insufficient. It is better to provide some experiment results of using RNN and transformer for the trend component.

Author response:
Using RNN and Transformer for the trend representation module
    - In appendix, we added the experiment results using the RNN and transformer architecture for trend representation module. The experiment results indicate that utilizing FC layer for trend component and CNN for seasonality component shows the best performance. Furthermore, results also confirm that our assumption that RNN, FC, and transformer layer is more suitable for capturing the trend characteristics compared to CNN is correct.

Reviewer comment (Novelty):
The term *Learning to Generate Predictor* is a little bit overstated from my perspective. My first impression would be a meta-learning model is considered. However, after reading the paper and codes. It seems just a usage of bilinear-type layer for me. I appreciate applying the bilinear layer since it seems not being used in recent forecasting literature. But *Learning to Generate Predictor* may not be the best term to summarize the model novelty for me. If the authors still prefer using *Learning to Generate Predictor*, it would be better to add more discussion to clearly state the difference from the meta-learning type model.

Author response:
- Relation between meta-learning type model and LGPred
    - Reviewer mentioned that the LGPred is a bilinear type layer rather than a meta-learning type model. We respectfully disagree. In general, meta-learning type models adopt a two-tier learning; the inner loop that conducts the adaptation to the given task and the outer loop that trains the model to better adapt to the given task. For our LGPred, the predictor generation path works as the inner loop conducting the adaptation to the given time series, and the training of the model works as the outer loop. In this sense, the work of LGPred is a kind of meta-learning type framework, rather than a bilinear structure. We draw inspiration from meta-learning model, and we make this point clearer in the introduction section of our updated paper. While applying the predictor originated from the input time-series to the input time-series bears some resemblance to bilinear structure, we believe the term ‘learning to generate predictor’ better explains the novelty of our LGPred.

Reviewer comment (Novelty):
The statement "*LGPred is the first attempt at adaptively generating a predictor reflecting the characteristics of each time series.*" seems also a little bit overclaimed. For example, in DeepAR, the network will first generate the $\mu$/$\sigma$ or $\mu$/$\alpha$ for Gaussian distribution or negative binomial (NB) distribution respectively. During the inference stage, the forecasting point will be sampled from the Gaussian/NB distribution. In this case, the Gaussian/NB distribution can be viewed as the *Predictor*, and the parameters in the predictor are learned with a network.

Author response:
- Learning to generate in prior work DeepAR
    - In DeepAR, the mean and standard deviation of prior distribution is predicted, and the prediction values are probabilistically generated from the distribution. However, considering the fact that the objective of DeepAR is probabilistic forecasting, the mean and standard deviation are the prediction output itself rather than the predictor. In this sense, we believe that our statement “LGPred is the first attempt at adaptively generating a predictor reflecting the characteristics of each time series” is not overclaimed.

Reviewer comment (Experiments):
The test data loader sets `drop_last = True`. In this case, the last several test samples are ignored, which will impact the accuracy of results in Table 1- Table 3. It would be better if the authors could fix it.

Author response:
- ‘drop_last’ config in test loader
    - Before answering, we are very impressed that the reviewer kindly checked out our code. The reviewer kindly informed that the drop_last config is set to be True, and this results in some test samples being ignored. We agree with the reviewer’s point. However, all prior works evaluate their methods under the same situation, and we follow the experimental setup for fair comparison. Please see the following official implementation provided by the authors.
        - Informer: https://github.com/zhouhaoyi/Informer2020/blob/main/exp/exp_informer.py (L77)
        - Autoformer: https://github.com/thuml/Autoformer/commit/d9100709b04e3e8361170794eba4f47b1afb217f (L20, committed after publish)
        - FEDformer: https://github.com/MAZiqing/FEDformer/blob/master/data_provider/data_factory.py (L19)
        - TimesNet: https://github.com/thuml/Time-Series-Library/blob/main/data_provider/data_factory.py (L28)
        - PatchTST: https://github.com/yuqinie98/PatchTST/blob/main/PatchTST_supervised/data_provider/data_factory.py (L19)
        - ETSformer: https://github.com/salesforce/ETSformer/blob/main/data_provider/data_factory.py (L19)
        - LTSF-Linear: https://github.com/cure-lab/LTSF-Linear/commit/6fe4c28ff36b4228792f2bbe513e807577e4a57e (L20, committed after publish)

Reviewer comment (Experiments):
The main results in Table 1 - Table 3 are from the model after hyperparameter searching. I'm wondering if the authors can provide a sensitive analysis of the parameter choices to further highlight the robustness of the proposed model.

Author response:
- Random control and Hyperparameter sensitivity analysis
    - Thank you for the reviewer’s kind comment to improve the completeness of our paper. For the random control experiment and hyperparameter sensitivity analysis, we added the results in appendix B5 and B6. Note that some of our original hyperparameter settings in Table 1 turn out to be vulnerable to random initialization, and we updated some scores in Table 1. Regarding the experiments in Table 3, we will update the random control experiment results if accepted.

Reviewer comment (Novelty):
The paper seems to omit discussions on contemporary related works. Notably, the structure of the proposed trend representation module is almost the same as TSMixer [1]. Furthermore, TiDE [2] has previously delved into refining linear models specifically for time series forecasting. Given the architectural similarities between these MLP-based models, a more in-depth comparison and differentiation would enhance clarity.

Author response:
We appreciate the reviewer suggesting the interesting related works. However, we believe the contribution of TSMixer[1] and TiDE[2] is clearly different from our LGPred. These works mainly focus on utilizing the multivariate features, while our LGPred focuses on adapting a model to a given time series. Meanwhile, the architectural similarity between these TSMixer and the trend representation module of our LGPred is an interesting point, and we added the comparison in method and related work section. Regarding the performance comparison, we wish the reviewer to understand the performance comparison is not conducted in the paper since these works are concurrent with our work published in August and September.

Reviewer comment (Experiments):
The absence of comprehensive ablation studies leaves the intrinsic value of each component in the proposed method ambiguous. For instance, the ablation analysis in [1] revealed that simpler stacked linear models (i.e., TMix-Only) could rival the performance of the presented methodology. This raises questions regarding the necessity of LGPred's individual components.

Author response:
We already have presented the ablation study results in appendix B.1. We believe that the results in appendix B1 prove the value and necessity of the components of our LGPred.