Paper: TILDE-Q: A Transformation Invariant Loss Function for Time-Series Forecasting (ICLR-2024)

Reviewer comment (Theory):
The amplitude shifting loss seems not optimal for noisy time series data, which is almost all time series of interest. The MSE loss aims to estimate the expectation of the probability distribution of the time series, which is the optimal solution, while the amplitude shifting loss always introduces a bias so that the prediction is always above or below the noisy signal.

Author response:
Thank you for your valuable feedback. For clarification, we would like to point out that having a non-optimal solution is often better than an optimal solution as prior work shows [1, 2]. Ishida et al. [1] revealed that providing some “tolerance” for the loss function makes training easier, resulting in better generalization. We also want to emphasize that the noise-robustness of our softmax loss depends on the choice of the distance function. We described and emphasized these issues in Appendix A.

Reviewer comment (Evaluation):
Similar to Point 2, in many cases, the amplitude shifting loss will compete against the standard MSE loss. For some problems, MSE loss will be optimal and, for other cases, probably the amplitude shifting loss makes sense. However, there is no clear guidance on how to decide which one to use.

Author response:
We think that the amplitude shifting is not designed to perfectly replace the standard MSE. Rather, amplitude shifting can be used as a complementary technique (e.g.,  [1, 2]) that allows tolerance for the models that do not necessarily aim at zero distance.  As a result, it achieves greater generalization than MSE, therefore, $L_{a.shift}$ could be used for all the datasets. In Appendix C, we provide multiple qualitative examples to illuminate this issue.

Reviewer comment (Theory):
The Fourier loss makes much more sense than the amplitude shifting loss, as it will capture the periodicity of the time series data. Recently, many studies have shown that capturing periodicity is a key element in time series forecasting. However, the authors should elaborate on how they compute the Fourier coefficients. For example, if the signal is not periodic, the Fourier series does not converge, making it difficult to truncate the Fourier series. Deciding the dominant frequency is non-trivial. There is also a question about the theoretical reason for choosing $\sqrt{T'}$ to truncate the Fourier series.

Author response:
In our design of $L_{phase}$, we chose the dominant frequency with $\sqrt{T’}$, which is equivalent to the statistical significance (i.e., one standard deviation) on Fourier coefficients. Furthermore, $L_{phase}$ guarantees the minimum number of dominant frequencies as $\sqrt{T’}$ when handling non-periodic signals. This regularization works as a noise filter, which is helpful in practice (detailed information is available in Appendix A). For an in-depth validation of the noise-robustness of our method, we provide qualitative examples in Appendix C.2.

Reviewer comment (Theory):
The logic behind the structure of the loss function in (7) is unclear. Specifically, why is $L_{phase}$ grouped with $L_{shift}$, and why is $L_{amp}$ separate?

Author response:
We introduce $\alpha$ and $1 - \alpha$ for $L_{a.shift}$ and $L_{phase}$ since they complement each other in terms of periodicity and statistical differences (in mean value and variance). $L_{amp}$ acts as a supplement, so we set $\gamma = 0.01$.  Note that we have provided detailed information on the setting and our source code in Appendix B. We have also described the design rationale for each sub-loss, $\alpha$, and $\gamma$ in Appendix A.

Reviewer comment (Experiments):
It may be interesting to compare the results with DLinear (https://arxiv.org/pdf/2205.13504.pdf), which proposes a very simple linear model with trend and seasonality decomposition, potentially similar to the loss functions proposed.

Author response:
Thank you for your comment. You can check the experimental results of NLinear in Anonymized GitHub (link available in Appendix B). In short, in most cases, MSE and TILDE-Q show comparable performance. However, we find that in the case of the Weather and Exchange datasets, which are hard to forecast due to frequent fluctuation patterns, TILDE-Q achieves better performance. In the case of the Weather dataset, we observe 6% to 12% improvements for NLinear.

Reviewer comment (Theory):
I don't see how the softmax loss can satisfy your eq (1) requirement; the optimal solution where \( d(y_i, \hat y_i) = k \) will result in a non-zero \( \mathcal{L}_{a,shift} \). Could the authors clarify on how the softmax helps strengthen value shift invariance?

Author response:
Yes, $L_{a.shift}$ satisfies a requirement. In short, we reformulated the “ensuring equal gaps between all points” into “making a uniform distribution of the gaps (i.e., $\sum_i{p_{d_i}\log p_{d_i}}$),” which is equivalent to entropy maximization. Therefore, we used the Softmax function for $L_{a.shift}$ and $L_{a.shift}$ satisfies Eq. 1. A detailed explanation of the sub-loss design can be found in Appendix A.

Reviewer comment (Experiments):
How do you choose the dominant frequency-domain coefficients for \( \mathcal{L}_{phase} \)? Does regularizing "non-dominant" frequencies for noise robustness help in practice? How do you deal with the fact that time series are finite-length signals (and there could be boundary effects)?

Author response:
In our design, we chose the dominant frequency with $\sqrt{T’}$, which is equivalent to the statistical significance (i.e., one standard deviation) on Fourier coefficients. Furthermore, $L_{phase}$ guarantees the minimum number of dominant frequencies as $\sqrt{T’}$ when handling non-periodic signals. This regularization works as a noise filter, which is helpful in practice (detailed information is available in Appendix A). For the validation of our noise-robustness, we also provide qualitative examples in Appendix C.2.

Reviewer comment (Evaluation):
It would have been valuable to carry out an analysis and see how common these distortions are in the datasets you evaluate on. It is difficult to assess significance otherwise (other than looking at the aggregate metrics).

Author response:
As we described in Sec. 3.1, distortions are defined between two time-series, which means they are the gap between two similar time-series. We want to emphasize that distortion itself can appear between ground-truth time-series and prediction results regardless of the dataset. If we ignore the potential impact of the distortion and keep using an objective function, which is vulnerable to distortion, the model produces uninformative forecasting results, as shown in the top and middle parts of Fig. 1. For a more thorough explanation and evaluation of the model and cases, we provide qualitative results in Appendix C.

Reviewer comment (Reproducibility):
It appears that the authors did not provide the complete version of the paper, as the appendices are missing.

Author response:
Thank you for your comment.

However, we kindly remind you that we have already uploaded the appendix in our submission. Please check supplementary materials.
In the appendix, we provide 1) recent time-series forecasting models, including NSFormer (AAAI'22). We also plan to update the results with NLinear and DLinear. Furthermore, the appendix provides the detailed experimental setting, results, and ablation study.

We are looking forward to you checking the appendix and updating the review. We will make our best efforts to answer your questions and update the manuscripts with your feedback.

Reviewer comment (Experiments):
The experiments are not complete.

Author response:
Thank you for your comment. We feel sorry that you missed the appendices that we have provided in the original submission. To help you find the information that you are looking for, we explain the location of the information. 

 - The experimental results with recent time-series forecasting models, including NSFormer (AAAI’22) and NLinear (AAAI’23) are available in Appendix C and Anonymized Github (through a link provided in Appendix B), respectively. 

- We describe the detailed experimental setting in Appendix B. The experimental results and their analysis are in Appendices C.1– C.3, while the qualitative evaluation is in C.2.

Reviewer comment (Evaluation):
The experiments lack detailed analysis.

Author response:
Thank you for your comment. We feel sorry that you missed the appendices that we have provided in the original submission. To help you find the information that you are looking for, we explain the location of the information. 

 - The experimental results with recent time-series forecasting models, including NSFormer (AAAI’22) and NLinear (AAAI’23) are available in Appendix C and Anonymized Github (through a link provided in Appendix B), respectively. 

- We describe the detailed experimental setting in Appendix B. The experimental results and their analysis are in Appendices C.1– C.3, while the qualitative evaluation is in C.2.

Reviewer comment (Reproducibility):
The paper is not complete since some important appendices related to experiments are missing.

Author response:
Thank you for your comment. We feel sorry that you missed the appendices that we have provided in the original submission. To help you find the information that you are looking for, we explain the location of the information. 

 - The experimental results with recent time-series forecasting models, including NSFormer (AAAI’22) and NLinear (AAAI’23) are available in Appendix C and Anonymized Github (through a link provided in Appendix B), respectively. 

- We describe the detailed experimental setting in Appendix B. The experimental results and their analysis are in Appendices C.1– C.3, while the qualitative evaluation is in C.2.

Reviewer comment (Experiments):
I'm interested in seeing additional results using TILDE-Q on recent time series forecasting models, including NLinear, DLinear, Scaleformer, PatchTST, Depts, N-hits, and SCINet.

Author response:
Thank you for your comment.

However, we kindly remind you that we have already uploaded the appendix in our submission. Please check supplementary materials.
In the appendix, we provide 1) recent time-series forecasting models, including NSFormer (AAAI'22). We also plan to update the results with NLinear and DLinear. Furthermore, the appendix provides the detailed experimental setting, results, and ablation study.

We are looking forward to you checking the appendix and updating the review. We will make our best efforts to answer your questions and update the manuscripts with your feedback.

Reviewer comment (Evaluation):
More in-depth analysis of the primary results are expected.

Author response:
Thank you for your comment. We feel sorry that you missed the appendices that we have provided in the original submission. To help you find the information that you are looking for, we explain the location of the information. 

 - The experimental results with recent time-series forecasting models, including NSFormer (AAAI’22) and NLinear (AAAI’23) are available in Appendix C and Anonymized Github (through a link provided in Appendix B), respectively. 

- We describe the detailed experimental setting in Appendix B. The experimental results and their analysis are in Appendices C.1– C.3, while the qualitative evaluation is in C.2.

Reviewer comment (Evaluation):
Improvements often seem to be marginal only (even though the score in a metric does not necessarily correlate with the actual quality of the qualitative forecast). Since you report that 10 models were trained, it would be easier to assess the loss quality if you provide error variations in form of $\sigma$ scores.

Author response:
We uploaded the results with the $\sigma$ score in our Anonymized GitHub, which is also provided in Appendix B. Please note that we only provide $\sigma$ scores in GitHub due to their large size. Additionally, we have updated various qualitative results and their analysis in Appendix C.

Reviewer comment (Evaluation):
Given the various computations required to calculate the TILDE-Q loss, it is unclear how much computational overhead the method introduces and whether the benefits outweigh and justify the increased time complexity.

Author response:
We updated our design rationale and discussion of time complexity in Appendix A. In short, TILDE-Q only requires O(n log n) time for its computation, which has an almost negligible impact on the overall time complexity.

Reviewer comment (Evaluation):
Could TILDE-Q serve as a performance measure itself too? The challenge of quantifying the skill of a time-series forecast is ubiquous. If the TILDE-Q score matches closely with visual inspections, this would be agreat metric as well. It would be a great outcome, if the judgement of subjects on the similarity of two time-series would correlate strongly with TILDE-Q.

Author response:
Yes. We think that if TILDE-Q is correctly used, it could work as a performance measure, as we show in the results, which will create a major impact on the domain. In addition, researchers could formulate their own measurements if their measurements met the requirements we described in Sec. 4.1.

Reviewer comment (Evaluation):
Inspecting the qualitative results in the appendix, can you formulate situations in which TILDE-Q performs poorly (consistently)?

Author response:
Thank you for your comment. We updated Appendix C, adding descriptions of the current stage of TILDE-Q with qualitative examples.