Paper: Learning to Embed Time Series Patches Independently (ICLR-2024)

Reviewer comment (Presentation):
In Table 2, does 46 mean 46 hours of training time?

Author response:
**Q1.  Unit of training time**

We apologize for the unclear expression. The unit of training time **Table 2 (of old version)** was sec/epoch, i.e., the time spent for one epoch of training in seconds. In the extended version of **Table 13 (of new version)**, we changed it to the total training time in minutes.

Reviewer comment (Experiments):
Have you conducted comparisons between the proposed method and alternative baselines, such as autoencoders or other self-supervised learning approaches tailored for time series data?

Author response:
**Q3. Comparison with self-supervised learning approaches other than MTM and CL**

Unfortunately, we could not find other self-supervised learning approaches competitive to MTM and CL in recent works.

We note that our ablation study in **Figure 4** includes **the comparison with the vanilla autoencoder, i.e., the one "w/o Dropout"**. However, following the conventional knowledge, autoencoders can learn good features when the size of the hidden dimension is small; otherwise, autoencoders might learn the trivial solution. **Figure 4** shows that the vanilla autoencoder is generally not better than ours as a pretext task.

We agree that exploring or revisiting other self-supervised learning approaches would be an interesting future work.

Reviewer comment (Evaluation):
Have you taken into account the limitations of the proposed model? Could you elaborate on potential challenges or specific scenarios in which the method might exhibit suboptimal performance?

Author response:
**Q4. Explanation of limitations and potential challenges**

Thanks for bringing this to our attention. Though our extensive experiments demonstrate the effectiveness of the proposed PI strategies across multiple benchmark datasets that we have evaluated, this observation might not be extendable to some cases.

For example, in our opinion, if a dataset has long-term dependencies, the dataset size is sufficiently large to learn such dependencies, and capturing the dependency is crucial for a downstream task, then a sophisticated PD strategy may outperform PI.

In fact, the PD architecture can be seen as a generalization of the PI architecture, as learning to ignore patch-dependence completely in the PD architecture makes it operate as like PI. Hence, the PD architecture should be able to replicate the performance of the PI architecture given that other conditions are ideal (though we could not empirically find such ideal cases).

We hope our proposed PI strategies provide a strong baseline for future TS representation learning works.

Reviewer comment (Evaluation):
The training and inference efficiency analyses are brief or missing. It would be useful to see whether the patch-independent design can also bring benefits to inference time.

Author response:
**W1. The training and inference efficiency analyses are brief or missing**

We agree that the information provided in **Table 2 (of old version)** was somewhat brief, so we extend and move it to **Table 13  (of new version)**. To highlight the additional information, **Table 13  (of new version)** now compares the inference time and performance. **PITS shows ~2.3x faster inference time than PatchTST**, thanks to the efficient PI architecture. Also, we **differentiate our method with or without CL**, such that **Table 13  (of new version)** also shows how much the pretraining time increases to achieve better performance when incorporating the naive or hierarchical CL.

Regarding the concern on **"training time compared to other supervised learning methods"**, we compare the training time of PITS and its supervised learning version (used the PI arch but no pretraining with the PI task)  in **Table I.1 (of new version)**. Self-supervised PITS pretrains the model for 100 epochs, trains the linear head for 10 epochs, and fine-tunes the entire model for 20 epochs in sequence, and supervised PITS trains for 100 epochs to achieve good performance. Given that pretraining is already done, fine-tuning the pretrained model is **x3~4 faster** than training from scratch, while providing better performance.

Reviewer comment (Experiments):
The model requires contrastive learning and reconstruction loss, which might drastically increase training time compared to other supervised learning methods. Therefore it would be useful to see it compared to supervised learning methods as well.

Author response:
**W1. The training and inference efficiency analyses are brief or missing**

We agree that the information provided in **Table 2 (of old version)** was somewhat brief, so we extend and move it to **Table 13  (of new version)**. To highlight the additional information, **Table 13  (of new version)** now compares the inference time and performance. **PITS shows ~2.3x faster inference time than PatchTST**, thanks to the efficient PI architecture. Also, we **differentiate our method with or without CL**, such that **Table 13  (of new version)** also shows how much the pretraining time increases to achieve better performance when incorporating the naive or hierarchical CL.

Regarding the concern on **"training time compared to other supervised learning methods"**, we compare the training time of PITS and its supervised learning version (used the PI arch but no pretraining with the PI task)  in **Table I.1 (of new version)**. Self-supervised PITS pretrains the model for 100 epochs, trains the linear head for 10 epochs, and fine-tunes the entire model for 20 epochs in sequence, and supervised PITS trains for 100 epochs to achieve good performance. Given that pretraining is already done, fine-tuning the pretrained model is **x3~4 faster** than training from scratch, while providing better performance.

Reviewer comment (Reproducibility):
What is the input horizon (look-back window/length of historic sequence) for the TSF task (Table 3)? This detail is missing.

Author response:
**Q1. Size of input horizon for the TSF task (Table 3 of old version; 2 of new version)**

We tuned the input horizon size with 3 candidates via validation: $L \in \\{336, 512, 768\\}$, where we added the information about validation in **Section 4.2**.

We realized that the detail on experimental settings is missing, so we added **Appendix B (of new version)** to provide them. We also clarify that we will release the code upon acceptance for reproducibility. However, if you find some hyperparameter or setting is still not clear, please feel free to share it.

Reviewer comment (Experiments):
Are the numbers in Table 3 averaged over multiple runs (with different random seeds) or are they single runs only?

Author response:
**Q2. The number of runs for Table 3 (of old version; 2 of new version)**

We ran the experiments 3 times with different random seeds, and they gave us consistent results: the standard deviation is mostly in the order of $10^{-4}$ to $10^{-3}$, while we observe the performance gain in the order of $10^{-2}$ to $10^{-1}$. For more details, we provide the standard deviation of the performance of PITS in **Appendix L (of new version)**.

Reviewer comment (Novelty):
Missing reference to previous method that Combine CL and MAE but in a different context and different method: Gong, Yuan, et al. "Contrastive audio-visual masked autoencoder." arXiv preprint arXiv:2210.07839 (2022).

Author response:
**W1-b, W2. References for combining CL and MM**

Thanks for the suggestion, we added a brief discussion on recent works on combining CL and MM in various domains [A,B,C,D,E]. We initially omitted them as our main focus is on PI strategies, but we agree that discussion on those works is indeed interesting and helpful to remind audiences of recent progress.

Among them, we note that SimMTM [E], which is a concurrent work to ours, proposed an MM task with a regularizer in its objective function in the form of a contrastive loss (though they did not claim that the regularizer is about contrastive learning but "series-wise similarity learning" and "point-wise aggregation"). This regularizer enforces that the original TS and its masked TS exhibit close representations and are distinctly different from representations of other series. However, it differs from our work in that it performs CL *between TS*, while our proposed CL operates *within a single TS*. Nevertheless, since our primary emphasis is on PI tasks and architectures, we removed the claim in **Section 1** that "our work is the first to integrate MTM and CL."

[A] Jiang, Ziyu, et al. "Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations." ICLR (2023).

[B] Yi, Kun, et al. "Masked image modeling with denoising contrast." ICLR (2023).

[C] Huang, Zhicheng, et al. "Contrastive masked autoencoders are stronger vision learners." arXiv preprint arXiv:2207.13532 (2022).

[D] Gong, Yuan, et al. "Contrastive audio-visual masked autoencoder." ICLR (2023)

[E] Dong, Jiaxiang, et al. "SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling." NeurIPS (2023).

Reviewer comment (Presentation):
Figure 5 is not as clear what it is conveying? What is Fig. 5 Left showing in terms of colours? Fig. 5 Right it is a bit confusing since it is PD-PI and x & y axis represent the slop and amplitude of diff between training and test phases, It is not clear to me what is exactly being changed to have the distribution shift in the time-series data?

Author response:
**Q1. About Figure 5**

The purpose of **Figure 5** is to see ***how robust PI task is under distribution shifts than PD task***. For better understanding, we added further details regarding the explanation of **Figure 5** in **Section 5**.

**[Left Panel]**

The **left panel of Figure 5** depicts a toy dataset comprising 98 time series exhibiting varying degrees of distribution shift, where **we assign a different color to each time series**. The degree of shift is characterized by changes in slope (trend) and amplitude (seasonality).
About how the toy dataset is exactly made, if the slope and amplitude of the training data are denoted as $s$ and $a$, respectively, then the slope and amplitude of the test data are represented as $r_1 \times s$ and $r_2 \times a$, respectively. Consequently, $r_1$ and $r_2$ serve as indicators of the slope and amplitude ratios, reflecting the degree of distribution shift.

**[Right Panel]**

The **right panel of Figure 5** illustrates the difference in MSE between the model trained with PI and PD tasks using the 98 time series depicted in the left panel (i.e., MSE of PD task - MSE of PI task). The x-axis and y-axis correspond to $r_1$ and $r_2$, respectively. For each $r_1$ and $r_2$, the color intensity shows how much PD is worse than PI (the darker, the worse.)

The observations from this figure are:
- 1. As the MSE difference (PD-PI) is non-negative for all cases, PI is more robust to distribution shifts than PD. Basically, PD becomes worse than PI as the change of $r_1$ and $r_2$ is larger (out of the center marked with the black rectangle).
- 2. The color gradient shows in which direction PD becomes even worse: when the slope is flipped (right to left) and the amplitude is increased (top to bottom).

If you have further questions, please feel free to share them. We are happy to address them and improve the clarity of our work.