From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This submission has been assessed by five knowledgeable reviewers. Two of them rated this paper as acceptable, one as marginally acceptable, while the other two reviewers voted for rejection (one reject and one strong reject). The supportive comments highlight clarity of presentation and good empirical performance on the benchmark data selected. The negative comments bring up several limitations, including limited choice of benchmark data, limited novelty in general (amplitude, phase and shift based loss components have been extensively studied in time series forecasting literature), missing or incomplete ablation study leaving questions of sensitivity of the proposed loss metric to hyperparameter value choices unclear, some assumptions being made without sufficient justification, and somewhat unclear motivation (reviewers suspect that only a subset of applications of time series forecasting may benefit from or truly focus on high fidelity reconstruction of the shape of the forecast waveforms, but the authors do not clarify this cornerstone aspect of the practical utility of their method). In summary, this work is interesting and has good potential. But at this point, its limitations outweigh its advantages, and as such it ranks slightly below the acceptance threshold for ICLR.  
justification_for_why_not_higher_score: Limitations of the work as currently presented, highlighted by the reviewers, prevent this paper from being a clear accept for ICLR 2023. 
justification_for_why_not_lower_score: n/a

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Response: additional justification and real-world examples
comment: Thank you for your comment. We provide more detailed justification and needs for the temporal dynamics. 

In this response, our main points are: 1) temporal dynamics is a fundamental characteristic of a time-series including periodic and non-periodic information; 2) many real-world examples like economics and traffic forecasting benefit from achieving shape awareness; 3) the critical information with periodic and non-periodic patterns is hardly modeled by Lp norm-based distance because Lp norm-based distances are vulnerable to distortions that trick the measurements. Consequently, temporal dynamics can be a key metric to evaluate forecasting performance. Below are the detailed answers for each comment.


> It is still not clear why temporal dynamics is a key metric to evaluate forecasting performance


As mentioned in Sec. 4, temporal dynamics (i.e., shape) is the informative periodic and non-periodic patterns in time-series data, including rise, drop, trough, peak, and plateau. The temporal dynamics includes non-periodic patterns such as sudden rising and dropping, which not only are hard to forecast but also not able to depict with Lp norm-based metrics [9]. In addition, temporal dynamics is well-known and one of the core terms in time-series data [1, 2].  Consequently, shape and its awareness have been discussed in many papers because they are fundamental characteristics of time-series [1, 2]. Shape awareness has always been one of the main topics for time-series forecasting [1-9]. Moreover, there are many real-world problems that benefit from shape awareness. However, Lp norm-based metrics, including MSE and RMSE, are hard to model such shape as loss functions, because it is vulnerable to distortions [1-9].

In time-series data mining, the shape is an inevitable factor for their analysis. Esling et al. [1] mentioned that “The purpose of time-series data mining is to try to extract all meaningful knowledge from the shape of data,” and there are many measures for shape-awareness, including DTW and LCSS. We list some papers about these various measures [2–9]. We believe that temporal dynamics should be a key metric for forecasting performance since it is a fundamental characteristic including a hard condition that should be carefully evaluated as well as plays a critical role in other papers as a key factor.

We additionally give some real-world examples that gain benefits from shape awareness. In 2007, Song et al. [10] surveyed economic forecasting methods – in their survey, the more recent works consider more shape-related features, such as periodicity and seasonality-trend decomposition. Furthermore, in their paper, they emphasize the importance of seasonality analysis and measuring the impact of sudden events – both are weak points of MSE-based models, as shown in Fig. 1 and DILATE [9] mentioned.

Traffic forecasting is another excellent example. In traffic forecasting, accurate forecasting for rush hour and traffic events (e.g., congestion) is highly valuable [11-13]. More precisely, the rising and dropping of the speed caused by a rush hour or events are worthy of real-world applications, such as navigation [14].


> Why not RMSE? Why not the maximum absolute difference?


Widely utilized Lp norm-based metrics, including MSE and RMSE, are incapable of handling the shape, such as periodicity, rising, or dropping, because these metrics have no transformation invariances necessary to model the shape [1-9]. As a result, these metrics score the forecasting results regardless of their potential information (e.g., periodic and non-periodic patterns). It causes information loss on shape, as shown in Fig. 1. In contrast, when we choose DTW, TDI, or LCSS, which consider temporal dynamics, they can accurately evaluate whether prediction preserves temporal features that are beneficial to assess the situation with periodicity, peak, or plateau. Using both MSE and temporal dynamics-related metrics could reinforce the evaluation qualities such as numerical differences, periodicity, etc. As a result, we insist that temporal dynamics is a key metric for the time-series evaluation.


> Besides, to define distortion properly, what is the assumption of transformation H?


In this paper, we assumed that for each distortion, transformation H is simple – in short, it could be represented by one basic operator – such as adding, multiplying, or function composition. For example, if there is some transformation H with $\mathcal{H}(\mathbf{F}) = c\mathbf{F} + k$, we reformulated them as $\mathcal{H}_1(\mathcal{H}_2(\mathbf{F})) = \mathcal{H}_1(c\mathbf{F}) = c\mathbf{F} + k$, and call $\mathcal{H}_1$ or $\mathcal{H}_2$ represents distortion.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: References
comment: 
[1] P. Esling and C. Agon, “Time-Series Data Mining,” in ACM Computing Surveys, 45(1), 2012

[2] J. Paparrizos and L. Gravano, “K-shape: Efficient and Accurate Clustering of Time Series,” in Proceedings of the ACM SIGMOD International Conference on Management of Data, 1855–1870, 2015.

[3] L. Chen and R. Ng. “On the marriage of Lp-norms and edit distance,” In VLDB, 792–803, 2004.

[4] L. Chen et al., “Robust and fast similarity search for moving object trajectories,” in Proceedings of the ACM SIGMOD International Conference on Management of Data, 491–502, 2005.

[5] Y. Chen et al., “Spade: On shape-based pattern detection in streaming time series,” in Proceedings of International Conference on Data Engineering, 786–795, 2007.

[6] C. Faloutsos et al., “Fast subsequence matching in time-series databases,” in Proceedings of the ACM SIGMOD International Conference on Management of Data, 419–429, 1994.

[7] E. Frentzos et al., “Index-based most similar trajectory search,” in Proceedings of International Conference on Data Engineering, 816–825, 2007

[8] M. Vlachos et al., “Discovering similar multidimensional trajectories,” in Proceedings of International Conference on Data Engineering, 673–684, 2002.

[9] V. Le Guen and N. Thome., “Shape and time distortion loss for training deep time series forecasting models,” in Advances in Neural Information Processing Systems, volume 32, 2019

[10] Song et al., “Tourism demand modeling and forecasting–A review of recent research,” in Tourism Management, 29, 2, 203–220, 2008.

[11] Y. Li and C. Shahabi., “A brief overview of machine learning methods for short-term traffic forecasting and future directions,” SIGSPATIAL Special, 10(1), 3–9, 2018.

[12] Y. Li et al., “Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,” In Proceedings of the International Conference on Learning Representations, 1–16, 2018.

[13] C. Lee et al., "A Visual Analytics System for Exploring, Monitoring, and Forecasting Road Traffic Congestion," in IEEE Transactions on Visualization and Computer Graphics, 26(11), 3133-3146, 2020.

[14] S. Jin et al., “A Visual Analytics System for Improving Attention-based Traffic Forecasting Model,” in IEEE Transactions on Visualization and Computer Graphics, 2022.

From: ICLR.cc/2023/Conference/Paper5710/Reviewer_4GPm
title: Keep the same score
comment: Thanks for your response.

For 1 and 2, it is still not clear why temporal dynamics is a key metric to evaluate forecasting performance. Why not RMSE? Why not the maximum absolute difference? When a new metric is proposed, you have to justify its necessity and benefit. 

Besides, to define distortion properly, what is the assumption of transformation H?

I would like to keep the same score.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Short Revision Summary
comment: We would like to appreciate the thoughtful and detailed comments by the reviewers. After revision, we changed and updated below:

 - Updated Figure 1 - Especially, we replaced the signals to have a more clear motivation.
 - We conducted more experiments with an additional model (i.e., FEDformer) and datasets (i.e., Exchange, Traffic, and Weather)
 - Added ablation study for the gamma.
 - Updated uncertain definitions and methods and provided real-world examples that need temporal dynamics (i.e., shape) modeling.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Respond to Official Review by Reviewer 4GPm
comment: We sincerely thank you for your detailed and helpful comments. Here are our revision lists for each weakness:

> 'The output Y_hat should have similar temporal dynamics with Y.' This is a strong assumption without much justification. It will be helpful to elaborate on why temporal dynamics are important and give examples of how they benefit time series forecasting in reality.

Thank you for your comment. For example, in traffic forecasting or economics prediction, it is much more valuable to forecast the temporal dynamics, such as rise, drop, peak, and plateau. Also, in previous research on time-series data mining [1], they mentioned one of the core parts is “measuring temporal dynamics.” We add some explanations about that goal, including two real-world examples and a related article. 


[1] Philippe Esling and Carlos Agon. Time-series data mining. ACM Computing Surveys, 45(1), 2012

> The definition of temporal dynamics, and distortion is quite fuzzy. It is necessary to give mathematical definitions for these two concepts.

Temporal dynamics are hard to be explained in terms of mathematics – it is an intuitive term to describe various features, such as periodicity, drop and rise, and peak. However, we can clarify distortions. Distortions could be described by the transformation that changes time-series but maintains a part of temporal dynamics. Thus, we revised the definition of distortion as follows:
Given two time-series $\mathbf{F}$ and $\mathbf{G}$ having a similar shape but not being mathematically identical, $\mathbf{F}$ could be formulated by transformation $\mathcal{H}(\mathbf{G})$. Then, we can call time-series $\mathbf{F}$ and $\mathbf{G}$ have a distortion, which could be represented by transformation $\mathcal{H}$.


> The authors criticized that DILATE handles time scaling with DTW and phase shifting with TDI but did not explain the link between DILATE and the new loss function. It will be good to have a separate section to clearly explain the relationship between DILATE and TILDE-Q. Is it beneficial to merge these metrics into one?

Actually, the goal of the DILATE differs from our loss function, TIDLE-Q. DILATE mainly depends on DTW which is designed to handle local time scaling, a special case of dynamic time scaling. On the other hand, TIDLE-Q aims to solve three distortions, amplitude shifting, phase shifting, and uniform amplification.
In Section 3.3, we mentioned that dynamic time scaling is inadequate for time-series forecasting because it allows dynamic alignment of the time-series. Specifically, DTW allows dynamic time scaling on pre-defined windows. In such windows, DTW could align the signal regardless of their information, for example, periodicity. As a result, the model makes misbehavior that could cheat DTW within the window, as shown in Fig. 1. 

In summary, DILATE highly relies on the DTW, which mainly focuses on dynamic time warping, and it is improper for our design rationale described in Section 3.3. Instead of making a new section, we added the possible misbehavior of DTW-based dynamic time warping invariance in Section 3.2.

> The Fourier transform is one of the classical ways to analyze time series data and the authors used the Fourier coefficients for phase shifting. However, it is not clear how to define dominant frequency.

For time-series forecasting, we commonly apply z-score normalization. As a result, the most of time-series are in a standard normal distribution. In a such condition, we defined dominant frequency with two conditions. For each frequency, we compute the norm of the Fourier coefficients, like $|c_n|^2$. Then, we filtered them with sequence length. But in some cases, such as Exchange dataset, there are few coefficients that satisfy the aforementioned condition, so we guarantee at least $\sqrt(T')$ number of dominant frequencies. We add the detailed design of the TILDE-Q in Section 4.

> Ablation study is not complete. It is not clear how the authors tuned hyperparameters alpha and gamma to generate results in Tables 1-4 which is critical to justify the effectiveness of the new metric. Thoroughly tuning these parameters can help get better results than roughly tuning compared loss functions like DILATE. It is encouraging that the authors showed some results of tuning alpha in the Appendix but still no result w.r.t. gamma is available.

Thank you for your comments. For all experiments in Tables 1-4, we utilized alpha = 0.5 and gamma = 0.01. we mentioned them in the detailed experiment setting in Appendix A. Also, we conduct an ablation study with gamma and explain them in the Appendix.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Respond to Official Review by Reviewer szr1
comment: Thank you for the worthy comments. Here are the revised lists for your suggestions:

> The experiments are not compared with the state-of-the art methods, such as FedFormer, NHITS and etc.

Thank you for the worthy comments. We additionally conduct experiments with FEDFormer and give extra results with other datasets, including Weather, Exchange, and Traffic.

> It's not sure that whether this method is general enough for other Transformer-based models, such as FedFormer and etc.

We additionally conduct experiments with FEDFormer and give results with other datasets, including Weather, Exchange, and Traffic. The experimental results reveal TILDE-Q is model agnostic and could be utilized from simple RNN-based structure to complicate transformer-based models.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Respond to Official Review by Reviewer wdgV
comment: Thank you for your constrcutive comments! We revised our paper with given suggestions:

> The paper does not study the loss components' weights α and γ. How sensitive are those hyper-params? What happens if we change them, qualitatively and quantitively?

We provide our ablation study results in the Appendix, including a study about α and γ. 


> TILDE-Q itself is not invariant to any of the amplitude shifting, phase shifting, and uniform amplification distortions.

TILDE-Q itself has no invariance for any of the three distortions, but it still has its ability to model the shape. Note that the most important fact of each subcomponent is complementary, not contradictory. Each subcomponent has its own tolerance such as an equal gap for amplitude shifting, but they share the common goal – modeling the temporal dynamics. We firmly believe that our design of each subcomponent is not contradictory, considering that we were able to enhance their benefits and overcome the drawback.


> Maybe it is worth studying what happens if we add MSE to TILDE-Q loss.

Thank you for your valuable suggestion. In this rebuttal phase, we focused on improving the reproducibility and quality of experiments. Even after rebuttal, we try to enhance our results, including N-HITS models for experiments or conducting more comparative analysis with MSE, DILATE, and TILDE-Q loss. We will leave additional results in openreview comments when we complete experiments about MSE and TILDE-Q.

> The paper claims that "time-series forecasting models should aim to both accurately forecast the value for each time-step and the prediction should have similar shapes as those in original time-series". This is not necessarily true. In practice, time-series forecasts deliver value by allowing future planning. Therefore, the cost of a specific error or distortion in the forecast depends strongly on the kind of planning we use the forecast for.


That’s true. "Time-series forecasting models should aim to both accurately forecast the value for each time-step and the prediction should have similar shapes as those in original time-series" is not necessarily true for all the tasks. As you mentioned, the cost of specific errors or distortions highly depends on the kind of planning. But in our paper, we make TILDE-Q handle the time-series regardless of the domain (or planning), so we assumed both of them should be aimed. We revised the terms in the paper.

Even though we utilize TILDE-Q to handle both accurate forecasting and shapes, we also believe that our study about distortions and corresponding invariances could help other future research or practical jobs.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Respond to Official Review by iGk7
comment: Thank you for your comments! The following are our answers for your review:

> Experimentation is not extensive; the authors are cutting by half the number of datasets from AutoFormer (Wu, 2021) experiments that used ETT, ECL, Exchange, Traffic, and Weather. Is it possible for only the datasets that align with the paper's research to be reported? Please report the results on all the missing datasets.

We additionally conducted our experiments on suggested datasets such as Exchange, Traffic, and Weather, adding the experiments of FEDFormer.

> Clarity Figure 1. needs to improve; the chosen example signal can be easily forecasted because it has evident seasonal patterns. The Figure suggests to the reader that MSE/normality forecasting assumption is incapable of accurate predictions for such a scenario; this intuition goes against a century of work on the topic. Please refer to ARIMA/ETS, and also consider changing the example signal; it gives the impression of creating a research problem that does not exist.

We clarified with a better example from a real-world dataset. We'd like to suggest and improve that "training models with MSE often fail to forecast accurately in a shape manner for difficult tasks (e.g., long-term forecasting).”

> Novelty Paper's novelty is limited. Phase, amplitude and shift subcomponents have been extensively studied. There are clear connections between the sub-losses and MSE please see: https://en.wikipedia.org/wiki/Parseval%27s_theorem

We agree with the argument that sub-losses and MSE have some connections between them, but we also want to say each sub-components have a different goal with MSE. 

First, we utilized Softmax-based distance, which has a totally different goal from MSE. In our Softmax-based distance, our loss function does not need to be zero Euclidean distance and does give some tolerance $k$. Furthermore, it optimizes to balance each point's relative gap, unlike the goal of MSE. This "balancing" technique enables models to have similar temporal dynamics.

Secondly, we utilized Fourier coefficients for the phase shifting components. However, using MSE is impossible to control the "dominant" frequency only. In the phase shifting components, our goal is 1) to catch the important periodicity with dominant frequency and 2) to get noise robustness with nondominant frequency. Based on Parseval's theorem, the sum of Fourier coefficients norm and the sum of the norm in Euclidean space are the same. However, after filtering, these are not identical. 

Moreover, as mentioned in FEDFormer [1], they utilized Fourier coefficients with sampling to build Fourier enhanced block (FEB) and Fourier enhanced attention (FEA), and improve Autoformer[2], which is very similar structures except for the FEB and FEA. Likewise, we firmly believe that we also have contributions and novelties in utilizing the sampling with dominant frequencies by outperforming and having different behaviors compared to MSE.


Lastly, for the uniform amplification components, we got an idea from the K-Shape clustering [3], which suggests better clustering methods than the DTW-based methods. For this one, we believe that our loss function has an equal contribution to k-shape because this is the first study to utilize them as a time-series forecasting objective function.


In short, our contribution has two stages: 1) the design rationale for the shape-aware loss function and 2) the loss function according to the design rationale. As we mentioned above, our loss function has different behaviors from MSE, contributing to the shape-aware loss function.

In addition, we suggest you an additional paper that has a clear connection with MSE. Ishida et al. [4] suggested the method named flooding, which criticizes that we do not need to have zero training error. The algorithm that this work utilize is simply a modified version of MSE. Also, please see other reviewers' comments that they believe that our paper is high quality and has high novelty.

We hope that the answers provided are sufficient for your review.


[1] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, 162, pp. 27268–27286, 2022


[2] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, volume 34, pp. 22419–22430, 2021.


[3] John Paparrizos and Luis Gravano. K-shape: Efficient and accurate clustering of time series. In ACM SIGMOD International Conference on Management of Data, pp. 1855–1870, 2015.

[4] Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. 2020. Do we need zero training loss after achieving zero training error? In International Conference on Machine Learning, Article 428, 4604–4614.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Respond to Official Review by Reviewer bzqM
comment: Thank you for your kind review! We will take an effort to enhance our contribution and conduct more experiments for the other applicable scenarios and future works.

From: ICLR.cc/2023/Conference/Paper5710/Authors
title: Respond to Official Review by Reviewer iGk7: reproducability
comment: 
> Reproducibility Given the quality of the experiments, reproducibility would be a second consideration, I would suggest to focus on improving them.

Thank you for your feedback. We additionally described our design in Section 4. Also, we will provide our code after acceptance.

From: ICLR.cc/2023/Conference/Paper5710/Reviewer_iGk7
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The authors propose the tilDEq loss function that combines phase, amplitude, and shift sub-losses and claims to outperform MSE. The paper needs to improve its motivation and experiments.
strength_and_weaknesses: Strengths
L-norm minimization for machine learning and in particular forecasting, is a very important topic.
The authors identify some challenges of L-norm minimization within the forecasting task.

Weaknesses
1. Experimentation is not extensive; the authors are cutting by half the number of datasets from AutoFormer (Wu, 2021) experiments that used ETT, ECL, Exchange, Traffic, and Weather. Is it possible for only the datasets that align with the paper's research to be reported? Please report the results on all the missing datasets.
2. Motivation of the paper is lacking, see comments of Figure 1.
clarity,_quality,_novelty_and_reproducibility: Clarity
Figure 1. needs to improve; the chosen example signal can be easily forecasted because it has evident seasonal patterns. The Figure suggests to the reader that MSE/normality forecasting assumption is incapable of accurate predictions for such a scenario; this intuition goes against a century of work on the topic. Please refer to ARIMA/ETS, and also consider changing the example signal; it gives the impression of creating a research problem that does not exist.

Novelty
Paper's novelty is limited. Phase, amplitude and shift subcomponents have been extensively studied.
There are clear connections between the sub-losses and MSE please see:
https://en.wikipedia.org/wiki/Parseval%27s_theorem

Reproducibility
Given the quality of the experiments, reproducibility would be a second consideration, I would suggest to focus on improving them.

summary_of_the_review: Strong reject.
The importance of the study of loss functions tailored for the forecasting task starkly contrasts with this submission's quality.
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper5710/Reviewer_bzqM
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work studied distortion handling in timer series forecasting problems. In particular, a novel loss function is proposed to guide models to consider amplitude, phase, and uniform amplification shifting invariance. 
strength_and_weaknesses: Strengths
1. The related work is comprehensive and shows a thorough analysis of different time series distortions.
2. The problem studied in this paper is important for the applications related to time series forecasting.
3. The paper is well-organized and easy to read.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, with high quality and high novelty. Furthermore, it could bring great value to the applications of time series forecasting. 
summary_of_the_review: This paper is well-written and provides a high-level of readability to most readers who work in the field of time series forecasting. The authors thoroughly explain why point-wise metrics could mislead the model and how models could benefit from learning shape-aware loss functions. The experiment shows the significant improvement the proposed loss function could bring, and a detailed experiment analysis is provided. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5710/Reviewer_wdgV
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a new loss function for time-series called TILDE-Q, which has 3 components. Each loss component is invariant to different distortions, including amplitude shifting, phase shifting, and uniform amplification. The paper starts with formal definitions of common time-series distortions and then designs 3 loss components that are invariant to 3 types of distortion. The experiment results suggest that the new loss function is model-agnostic. It can improve the accuracy of different existing time-series models, even on common existing metrics such as MSE or DTW.


strength_and_weaknesses: Strength:
  - The paper is self-contained. It provides a systematic study of different time-series distortions and builds up invariant loss components from there.
  - The paper is well-presented, with clear notations and figures. The appendix is also very useful with detailed experiment setup and qualitative results.
  - The new loss function yields significant accuracy improvements across different time-series models on various datasets.

Weaknesses:
  - The paper does not study the loss components' weights $\alpha$ and $\gamma$. How sensitive are those hyper-params? What happens if we change them, qualitatively and quantitively?
  - TILDE-Q itself is not invariant to any of the amplitude shifting, phase shifting, and uniform amplification distortions.
  - Maybe it is worth studying what happens if we add MSE to TILDE-Q loss.
  - The paper claims that "time-series forecasting models should aim to both accurately forecast the value for each time-step and the prediction should have similar shapes as those in original time-series". This is not necessarily true. In practice, time-series forecasts deliver value by allowing future planning. Therefore, the cost of a specific error or distortion in the forecast depends strongly on the kind of planning we use the forecast for.
clarity,_quality,_novelty_and_reproducibility: Even though the paper borrows some key ideas from prior works, such as normalized cross-correlation function or dominant Fourier coefficient, the TILDE-Q loss function construction is novel, as far as I can tell.
summary_of_the_review: The paper gives a good systematic review of common time-series distortions. The paper is well-presented with good support materials. The designs of the 3 loss components are elegant and effective. The experiment results are strong, which shows that TILDE-Q, the weighted sum of the 3 loss components, can improve the accuracy of different existing time-series models, even on common existing metrics such as MSE or DTW.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5710/Reviewer_szr1
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper investigates distortions in amplitude and phase to understand shape-awareness and distortion invariances in time-series forecasting. This paper uses TILDE-Q that has invariances to many existing distortions and achieves shape-awareness and informative forecasting in a timely manner. Experimental results show the efficacy of proposed method.
strength_and_weaknesses: Strengths:

1. It's novel to investigates distortions in amplitude and phase to understand shape-awareness and distortion invariances
2. The formulation on time series distortions in transformation perspective is novel and interesting. 

Weaknesses:

1. The experiments are not compared with the state-of-the art methods, such as FedFormer, NHITS and etc.
2. It's not sure that whether this method is general enough for other Transformer-based models, such as FedFormer and etc.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The method is novel and looks reproducible.
summary_of_the_review: This paper investigates distortions in amplitude and phase to understand shape-awareness and distortion invariances in time-series forecasting. This paper uses TILDE-Q that has invariances to many existing distortions and achieves shape-awareness and informative forecasting in a timely manner. Experimental results show the efficacy of proposed method.

However,  the experiments are not compared with the state-of-the art methods, such as FedFormer, NHITS and etc. Also it's not sure that whether this method is general enough for other Transformer-based models, such as FedFormer and etc.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: NO
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5710/Reviewer_4GPm
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors proposed a new loss function (transformation invariant loss function with distance equilibrium) for time series forecasting. This new loss function helps capture the shape of time-series sequences. The authors tested the effectiveness of the new loss function on multiple datasets and models.
strength_and_weaknesses: Strength:
1. The authors have done a thorough literature review on the SOTA methods, models, and metrics on time series forecasting.
2. The authors also have experimental results to compare the proposed loss function and the other two metrics with 4 SOTA models on 5 datasets.

Weaknesses:
1. 'The output Y_hat should have similar temporal dynamics with Y.'
This is a strong assumption without much justification. It will be helpful to elaborate on why temporal dynamics are important and give examples of how they benefit time series forecasting in reality.
2. The definition of temporal dynamics, and distortion is quite fuzzy. It is necessary to give mathematical definitions for these two concepts.
3. The authors criticized that DILATE handles time scaling with DTW and phase shifting with TDI but did not explain the link between DILATE and the new loss function. It will be good to have a separate section to clearly explain the relationship between DILATE and TILDE-Q. Is it beneficial to merge these metrics into one?
4. The Fourier transform is one of the classical ways to analyze time series data and the authors used the Fourier coefficients for phase shifting. However, it is not clear how to define dominant frequency.
6. Ablation study is not complete.
It is not clear how the authors tuned hyperparameters alpha and gamma to generate results in Tables 1-4 which is critical to justify the effectiveness of the new metric. Thoroughly tuning these parameters can help get better results than roughly tuning compared loss functions like DILATE. It is encouraging that the authors showed some results of tuning alpha in the Appendix but still no result w.r.t. gamma is available.
clarity,_quality,_novelty_and_reproducibility: 1. The writing needs to be improved significantly to better present the work.
2. The authors did not give enough details to reproduce all results in this work.
summary_of_the_review: The paper assembles too many strong assumptions without very good justification. It is a clear rejection.

-----------
I have read the authors' responses and would like to keep the same score.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough