Paper: RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies (ICLR-2024)

Reviewer comment (Evaluation):
Overall limitations of the approach are not discussed. When does it break? What implicit assumptions are made? What time series aspect one would need to consider to choose this approach over another in practice?

Author response:
**W2:** Hyper parameter tuning of RobustTSF is missing: it’s never discussed how \tau of 0.3 was chosen and Figure 5 suggests that choice of \tau would affect the comparison to the baselines.

**A:** The selection of $\tau$, denoting the filtering threshold akin to the threshold in loss-based sample selection for learning with noisy labels [R1], was fixed at 0.3 for all experiments. This choice was based on our observation that different values of τ around 0.3 did not significantly impact the performance of RobustTSF, as depicted in Figure 5. It's important to clarify that Figure 5 is not intended for comparing RobustTSF to other methods but rather to demonstrate the robustness of RobustTSF to variations in the hyperparameter $\tau$.

[R1] co-teaching: robust training of deep neural networks with extremely noisy labels

Reviewer comment (Evaluation):
No confidence bounds in the experimental results makes it hard to judge the significance of the evaluation results.

Author response:
**W3:** Overall limitations of the approach are not discussed. When does it break? What implicit assumptions are made? What time series aspect one would need to consider to choose this approach over another in practice?

**A:** We provide the limitation of our method below and has added the content in our revised version.

- Our method primarily focuses on point-wise anomalies, with limited exploration of sequence-wise anomalies through specific experiments. The theoretical analysis of sequence-wise anomalies remains somewhat unclear, as real-world sequence anomalies can be intricate. The statistical modeling and theoretical robustness against these anomalies present a challenging and unexplored aspect in our current work.
- Our theoretical analysis is grounded in a statistical perspective, aligned with robust learning theory in the Learning with Noisy Labels (LNL) domain. However, in cases of limited training size, a potential disparity might arise between theoretical analyses and empirical findings.

From the limitation, the training size may impact the performance of RobustTSF, given that the theoretical analyses of losses are from a statistical perspective. Therefore, an implicit assumption of RobustTSF is that the training size should not be very small; otherwise, applying RobustTSF may not surpass baselines. While the datasets used in the paper suggest that training size may not be a significant issue, we conducted additional experiments to explore the gap between the baseline and RobustTSF on the Electricity dataset (reporting best epoch MAE for each setting) concerning the training size under a Gaussian anomaly setting (anomaly rate = 0.3):


|           | 0.5% | 1%    | 2%    | 10%   | 50%   | 100%  |
| --------- | ---- | ----- | ----- | ----- | ----- | ----- |
| MAE       | 1.08 | 0.901 | 0.713 | 0.578 | 0.240 | 0.210 |
| RobustTSF | 1.08 | 0.913 | 0.606 | 0.418 | 0.181 | 0.177 |

The results demonstrate that RobustTSF outperforms baselines when the training size of the dataset is not very small.

Reviewer comment (Experiments):
Only two datasets of very similar properties are used for evaluation, which again limits the assessment of method applicability.

Author response:
**W4:** No confidence bounds in the experimental results makes it hard to judge the significance of the evaluation results

**A:** To show our results are statistically consistent,Specifically, we reran the pivotal experiments outlined in Table 2 of our main paper, focusing on the Electricity dataset. We present the best epoch MAE as the mean (standard deviation) for the three runs of each configuration, as follows:


| Electricity | Clean         | Const 0.1     | const 0.3     | Missing 0.1   | Missing 0.3   | Gaussian 0.1  | Gaussian 0.3  |
| ----------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| MSE         | 0.187(0.002)  | 0.200 (0.003) | 0.234 (0.003) | 0.227 (0.005) | 0.309 (0.015) | 0.192 (0.003) | 0.228 (0.002) |
| MAE         | 0.183(0.002)  | 0.191(0.004)  | 0.209 (0.004) | 0.207 0.003() | 0.223 (0.006) | 0.198 (0.003) | 0.210 (0.004) |
| Offline     | 0.184(0.003)  | 0.188 (0.003) | 0.217 (0.004) | 0.192 (0.003) | 0.213 (0.004) | 0.184 (0.002) | 0.191 (0.003) |
| Online      | 0.184(0.002)  | 0.199 (0.003) | 0.225 (0.003) | 0.194 (0.003) | 0.230 (0.003) | 0.195 (0.003) | 0.209 (0.002) |
| Loss sel    | 0.180 (0.005) | 0.205(0.003)  | 0.209 (0.004) | 0.201 (0.002) | 0.215(0.003)  | 0.196 (0.002) | 0.201 (0.003) |
| RobustTSF   | 0.177(0.002)  | 0.177 (0.002) | 0.184 (0.002) | 0.180(0.002)  | 0.195(0.001)  | 0.175(0.001)  | 0.177 (0.001) |


These results demonstrate the statistical consistency of our findings.

Reviewer comment (Theory):
Proposition 2: the statement is not clear: does it hold under the conditions of Theorem 1? If yes, then the statement is obsolete, if not, then where does C_x comes from?

Author response:
**Q1:** Proposition 2: the statement is not clear: does it hold under the conditions of Theorem 1? If yes, then the statement is obsolete, if not, then where does C_x comes from?

**A:** The definition of $C_x$ in both Theorem 1 and Proposition 2 remains consistent. However, it's crucial to note that these propositions address different scenarios. The condition outlined in Theorem 1 pertains to the assumption that $C_x$ is constant with respect to the  forecasting models. In Proposition 2, we explore loss functions where $C_x$ does not remain constant with respect to the  forecasting models. Specifically, Proposition 2 aims to demonstrate that if $C_x$ can be estimated, it allows us to normalize the loss, thereby enhancing its robustness. This insight draws inspiration from the field of learning with noisy labels in (image) classification, where techniques such as normalizing cross entropy have been employed to enhance robustness to label noise [R1].

[R1] normalized loss functions for deep learning with noisy labels

Reviewer comment (Writing):
Section 6.2: why notations change from z to x? It’s not clear what x is and how they are different from z.

Author response:
**Q2:** Section 6.2: why notations change from z to x? It’s not clear what x is and how they are different from z

**A:** As explained in our problem formulation in Section 3, for a given time series with anomalies, $\widetilde{z_i}$ is the i-th value and $\widetilde{\mathbf{x}}\_{i}$ is the i-th segment. The length of $\widetilde{\mathbf{x}}\_{i}$ is $\alpha$. This notation is consistently applied in Section 6.2.

Reviewer comment (Theory):
End of page 6: while RobustTSD does not need a pre-trained DNN, it relies on limited trend estimation precedure in eq. (4) that implicitly assumes the smooth behavior of the time series. I’d see this as a disadvantage that one will have to adapt eq. (4) in unclear way to different datasets, while DNN would adopt automatically via pre-training.

Author response:
**Q3:** End of page 6: while RobustTSD does not need a pre-trained DNN, it relies on limited trend estimation precedure in eq. (4) that implicitly assumes the smooth behavior of the time series. I’d see this as a disadvantage that one will have to adapt eq. (4) in unclear way to different datasets, while DNN would adopt automatically via pre-training.

**A:** We acknowledge the need for trend estimation in RobustTSF, and we agree that this estimation may not be accurate for every dataset. The main objective of our paper is to highlight the time efficiency of RobustTSF in comparison to DIR (detection imputation retraining pipeline), which relies on pre-trained DNNs. As demonstrated in [R1], DIR involves multiple steps, including training a DNN on noisy time series and then using this pre-trained DNN for anomaly detection. However, this multi-step process increases training time and may result in an inferior pre-trained DNN, especially when trained on noisy data.

[R1] Recurrent neural networks and robust time series prediction

Reviewer comment (Experiments):
Ablation study section: what is presented is not an ablation study, but rather a hyperparameter tuning on the test dataset. An ablation study would, for example, consider the effect of LNL and sample selection separately, i.e. what happens when one applies only LNL or only sample selection.

Author response:
**Q5:** Ablation study section: what is presented is not an ablation study, but rather a hyperparameter tuning on the test dataset. An ablation study would, for example, consider the effect of LNL and sample selection separately, i.e. what happens when one applies only LNL or only sample selection.

**A:** Thank you for your feedback. We have updated the section title in the main paper to "Hyperparameter Tuning" and conducted the suggested ablation studies. The results are presented in the table below:

| Electricity           | Const 0.1 | const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| --------------------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| Only LNL (loss)       | 0.191     | 0.206     | 0.205       | 0.225       | 0.199        | 0.210        |
| Only sample selection | 0.183     | 0.186     | 0.184       | 0.237       | 0.180        | 0.182        |
| RobustTSF             | 0.177     | 0.183     | 0.181       | 0.194       | 0.176        | 0.177        |

| Traffic           | Const 0.1 | const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| --------------------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| Only LNL (loss)       | 0.209     | 0.231     | 0.266       | 0.295       | 0.216        | 0.233        |
| Only sample selection | 0.199     | 0.220     | 0.224       | 0.310       | 0.196        | 0.210        |
| RobustTSF             | 0.198     | 0.203     | 0.200       | 0.243       | 0.185        | 0.202        |

These additional analyses have been included in the revised version of our paper.

Reviewer comment (Reproducibility):
Parameter \tau seems to be scale dependent and while appendix mentions that the time series are normalized, the main text omits this detail.

Author response:
**Q6:** Parameter \tau seems to be scale dependent and while appendix mentions that the time series are normalized, the main text omits this detail.

**A:** Thank you for pointing out the omission. We have addressed this by adding a clarification in the main paper. Specifically, the data is normalized to have a mean of 0 and a standard deviation of 1 before adding anomalies and training process. Additionally, it's worth noting that we consistently set the parameter $\tau$ to 0.3 for all configurations across different datasets. The normalized information has been included in the revised version of our paper.

Reviewer comment (Experiments):
I noticed that in your experimental section, you used a large input length to predict a relatively small output length. For example, in section 7.2, you used an input length of 96 to predict an output data of length [4, 8]. However, for time series forecasting tasks, the typical setting is an input length of 12 and an output length of 12. Therefore, I am unsure if your model is applicable to this setting. I recommend conducting an additional set of experiments where the input series has a length of 12 and the output series has a length of 12, in order to assess the performance of your model in this specific scenario.

Author response:
We appreciate your observation regarding the input and output lengths in our experiments. Following your suggestion, we conducted additional experiments on the Electricity dataset with an input length of 12 and an output length of 12 and report the best epoch MAE. The results are presented below:

|           | Const 0.1 | Const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| --------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| MAE       | 0.550     | 0.575     | 0.544       | 0.572       | 0.541        | 0.565        |
| MSE       | 0.551     | 0.578     | 0.545       | 0.595       | 0.545        | 0.568        |
| Offline   | 0.539     | 0.552     | 0.527       | 0.616       | 0.531        | 0.546        |
| Online    | 0.623     | 0.627     | 0.617       | 0.680       | 0.605        | 0.654        |
| Loss sel  | 0.560     | 0.564     | 0.535       | 0.623       | 0.549        | 0.563        |
| RobustTSF | 0.512     | 0.543     | 0.525       | 0.547       | 0.516        | 0.523        |

These results affirm that RobustTSF maintains good performance with input and output length of 12. We have included this new set of experiments in our revised version, and the corresponding code is provided in the supplementary material, named “RobustTSF_multi_step”.

Reviewer comment (Writing):
Please note the spelling error: "k-the" below Equation (5) should be corrected to "k-th."

Author response:
Thank you for pointing out the spelling error. We have corrected "k-the" to "k-th".

Reviewer comment (Theory):
In Theorem 1, why is it necessary for the anomaly rate to be less than 0.5?

Author response:
Regarding Theorem 1, the necessity for the anomaly rate to be less than 0.5 is derived from Appendix A.1, where the proof indicates that to align the loss on noisy time series labels consistently with clean labels, $1-2\eta$ must be greater than 0. Therefore, the anomaly ratio must be less than 0.5.

Reviewer comment (Evaluation):
In your method, you mentioned three types of anomalies, and your analysis of loss robustness indicates that MAE is robust to all three types of anomalies, while MSE is only robust to one of them. Can we understand from this that MAE is more suitable as a loss function than MSE in all situations? Are there any criteria to evaluate the robustness of other loss functions?

Author response:
Concerning the choice between MAE and MSE, our analysis indicates that MAE is more suitable, as it exhibits robustness to all three types of anomalies. Previous TSFA research mainly use MSE and our study shows that MSE is only robust to Gaussian anomalies. 
We acknowledge your observation and provide Proposition 2 as a criterion to evaluate other potential loss functions. If a loss function satisfies the conditions outlined in Proposition 2, it can be considered robust to Constant and Missing type anomalies. However, the exploration of additional criteria for the theoretical evaluation of alternative loss functions is outside the scope of this paper, and we consider it as a potential avenue for future research.

Reviewer comment (Presentation):
The first weakness of the article is the presentation of the main algorithm's idea. Its explanation is spread across several pages, and one has to wait until the 6th page to get an idea of how the algorithm will work, even though the idea itself is rather simple: identifying anomalies in the signal and applying very weak penalties to the cost in the corresponding region. It would be much easier to read the article if even a brief description were introduced very early on to understand the relevance of the different theorems formulated.

Author response:
**A:** We apologize for any confusion caused by the current organization. The structure aligns with the novelty of the task in time series forecasting with anomalies (TSFA), where formal definitions of problem formulations and anomaly types are crucial. Sections 4 and 5 are dedicated to analyzing the distinctions between TSFA and learning with noisy labels (LNL). We acknowledge the need for an early algorithm description and have addressed this by adding it to the Introduction section.

Reviewer comment (Novelty):
Regarding novelty, the main theorems seem to be adapted quite directly from other works, so there isn't much novelty there. Furthermore, the idea of filtering and training on robust windows is not very new either, and the filtering method is also pre-existing. Apart from adapting the framework for anomalies to time series, there isn't a significant novelty.

Author response:
**A:**  We clarify that the original robust loss theorem we followed was designed for classification in conventional LNL settings [R1]. TSFA introduces a regression problem with unique anomalies, necessitating the formal definition and examination of loss robustness specific to TSFA. To our knowledge, no previous studies address loss robustness in TSFA tasks.

While the concept of filtering has been explored in earlier TSFA studies [R2], it is essential to recognize notable distinctions. Previous filtering strategies, as observed in [R2], apply the same approach as in conventional LNL problems, without taking into account the nuances specific to TSFA. In contrast, our filtering strategy is meticulously crafted by considering the disparities between TSFA and LNL, resulting in consistent performance improvement across diverse anomaly types.

[R1] Robust loss functions under label noise for deep neural networks.

[R2] Robust learning of deep time series anomaly detection models with contaminated training data.

Reviewer comment (Novelty):
Regarding Section 5, which analyzes the effect of the position of anomalies on forecasting, it seems quite normal that anomalies near the end of the signal disrupt the prediction more than anomalies at the beginning. The system's state is disrupted, making forecasting more challenging. In contrast, if the anomalies are at the beginning, the model has time to stabilize on the 'correct' values.

Author response:
**A:**  Although the observation may appear commonplace, it serves as a foundational point of distinction for TSFA compared to LNL (Learning with Noisy Labels). This observation directs the algorithm design, enabling the selection of input time series where anomalies are sparse in the time series points located near the label. Leveraging sample robustness and loss robustness, the algorithm achieves robustness goals akin to general LNL tasks. Similar findings have been noted in temporal graph learning [R1], emphasizing the significance of recent edges for target predictions.

[R1] Adaptive data augmentation on temporal graph

Reviewer comment (Experiments):
However, the biggest weakness of the article lies in the experiments, which do not include any modern time series forecasting baselines (such as PatchTST, DLinear, Autoformer, ...). From a few quick experiments I conducted using the authors' code, DLinear seems to easily outperform the proposed model on the noisy data. This raises the question of the method's relevance if state-of-the-art models do not struggle with the anomalies considered by the authors. Of course, I may be mistaken, but in any case, it would be helpful to have the performance of state-of-the-art algorithms on this kind of problem to understand the challenges posed by the envisaged anomalies.

Author response:
**A:** Firstly, in the context of learning with noisy label tasks (LNL), it is conventional to maintain a consistent network structure for all methods to ensure a fair evaluation of robustness and superiority [R1, R2, R3] (eliminating network structure bias). Our paper proposes a model-agnostic method, necessitating the use of the same network structure for all methods. LSTM is chosen due to its prevalence in recent robust time series forecasting papers [R4, R5, R6], as well as in related works on TSFA tasks [R7, R8, R9].

While we acknowledge the importance of exploring other network structures, we demonstrate the versatility of RobustTSF in Appendix F.12 by showcasing improved performance with TCN and transformer models. Notably, PatchTST, Autoformer, and Pyformer are transformer-based models.

Secondly, concerning DLinear [R10], which is asserted to have superior performance to transformer-based models, we have thoroughly analyzed and experimented with it. Analytically, DLinear's reliance on linear regression for both trend and seasonality may limit its performance since the randomness of anomalies introduces challenges in trusting the seasonality of input time series. Moreover, Table 15 in our paper illustrates that relying solely on the trend yields inferior performance compared to RobustTSF.

Experimentally, we ran DLinear on the Electricity and Traffic datasets using the official code [R11], incorporating various anomaly types as detailed in Table 2. The results are summarized below (reporting best epoch MAE):
              
| Dataset: Electricity | Const 0.1 | Const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| -------------------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| Dlinear              | 0.176     | 0.189     | 0.179       | 0.198       | 0.195        | 0.347        |
| RobustTSF            | 0.177     | 0.183     | 0.181       | 0.194       | 0.176        | 0.177        |


| Dataset: Traffic | Const 0.1 | Const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| ---------------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| Dlinear          | 0.219     | 0.238     | 0.232       | 0.318       | 0.277        | 0.472        |
| RobustTSF        | 0.198     | 0.203     | 0.200       | 0.243       | 0.185        | 0.202    

These results illustrate that, although DLinear may exhibit a slight advantage in certain anomaly scenarios, RobustTSF consistently outperforms DLinear in the majority of settings, especially when confronted with Gaussian-type anomalies. The code for DLinear is available in the supplementary material under the filename "train_noisy_dlinear.py."


[R1] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels

[R2] Generalized cross entropy loss for training deep neural networks with noisy labels

[R3] Early-Learning Regularization Prevents Memorization of Noisy Labels

[R4]  Robust probabilistic time series forecasting.

[R5] Huber additive models for non-stationary time series analysis.

[R6] Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. 

[R7] Recurrent neural networks and robust time series prediction

[R8] Resilient neural forecasting systems

[R9] Robust learning of deep time series anomaly detection models with contaminated training data.

[R10] Are Transformers Effective for Time Series Forecasting? 

[R11] https://github.com/cure-lab/LTSF-Linear/blob/main/models/DLinear.py

Reviewer comment (Experiments):
Can you provide experiments with the usual baseline of the state of the art ? (PatchTST, Dlinear, Autoformer, Pyformer, ...)

Author response:
**A:** Firstly, in the context of learning with noisy label tasks (LNL), it is conventional to maintain a consistent network structure for all methods to ensure a fair evaluation of robustness and superiority [R1, R2, R3] (eliminating network structure bias). Our paper proposes a model-agnostic method, necessitating the use of the same network structure for all methods. LSTM is chosen due to its prevalence in recent robust time series forecasting papers [R4, R5, R6], as well as in related works on TSFA tasks [R7, R8, R9].

While we acknowledge the importance of exploring other network structures, we demonstrate the versatility of RobustTSF in Appendix F.12 by showcasing improved performance with TCN and transformer models. Notably, PatchTST, Autoformer, and Pyformer are transformer-based models.

Secondly, concerning DLinear [R10], which is asserted to have superior performance to transformer-based models, we have thoroughly analyzed and experimented with it. Analytically, DLinear's reliance on linear regression for both trend and seasonality may limit its performance since the randomness of anomalies introduces challenges in trusting the seasonality of input time series. Moreover, Table 15 in our paper illustrates that relying solely on the trend yields inferior performance compared to RobustTSF.

Experimentally, we ran DLinear on the Electricity and Traffic datasets using the official code [R11], incorporating various anomaly types as detailed in Table 2. The results are summarized below (reporting best epoch MAE):
              
| Dataset: Electricity | Const 0.1 | Const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| -------------------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| Dlinear              | 0.176     | 0.189     | 0.179       | 0.198       | 0.195        | 0.347        |
| RobustTSF            | 0.177     | 0.183     | 0.181       | 0.194       | 0.176        | 0.177        |


| Dataset: Traffic | Const 0.1 | Const 0.3 | Missing 0.1 | Missing 0.3 | Gaussian 0.1 | Gaussian 0.3 |
| ---------------- | --------- | --------- | ----------- | ----------- | ------------ | ------------ |
| Dlinear          | 0.219     | 0.238     | 0.232       | 0.318       | 0.277        | 0.472        |
| RobustTSF        | 0.198     | 0.203     | 0.200       | 0.243       | 0.185        | 0.202    

These results illustrate that, although DLinear may exhibit a slight advantage in certain anomaly scenarios, RobustTSF consistently outperforms DLinear in the majority of settings, especially when confronted with Gaussian-type anomalies. The code for DLinear is available in the supplementary material under the filename "train_noisy_dlinear.py."


[R1] Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels

[R2] Generalized cross entropy loss for training deep neural networks with noisy labels

[R3] Early-Learning Regularization Prevents Memorization of Noisy Labels

[R4]  Robust probabilistic time series forecasting.

[R5] Huber additive models for non-stationary time series analysis.

[R6] Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. 

[R7] Recurrent neural networks and robust time series prediction

[R8] Resilient neural forecasting systems

[R9] Robust learning of deep time series anomaly detection models with contaminated training data.

[R10] Are Transformers Effective for Time Series Forecasting? 

[R11] https://github.com/cure-lab/LTSF-Linear/blob/main/models/DLinear.py

Reviewer comment (Reproducibility):
The ranges of hyper parameters, noise scales, and stds, and other variables related to normalizing the data and running the algorithms need to be clearly identified.

Author response:
**A:** Thank you for your valuable suggestion. Regarding noise scales and standard deviations, these parameters are integral to our noise model, typically spanning the range of $(-\infty, +\infty)$. As detailed in Appendix E, our approach involves normalizing the training set to a mean of 0 and a standard deviation of 1, followed by the addition of anomalies. Specifically, for constant-type anomalies, the noise scale is $\epsilon = 0.5 \cdot \text{std} = 0.5$; for missing-type anomalies, the noise scale is $\epsilon = 0$; and for Gaussian-type anomalies, the noise scale is $\epsilon \sim \mathcal{N}(0,2\cdot \text{std})$. This choice is inspired by the initial paper on the TSFA task [R1]. In Appendix F.7, we illustrate how these noise modeling hyperparameters influence the performance of RobustTSF.

Concerning the hyperparameters necessary for algorithm execution, the primary parameters are $\lambda$ and $\tau$, both with a range of $(0, +\infty)$. To demonstrate RobustTSF's robustness to hyperparameters, we consistently employ the same values across all settings and datasets in our experiments. In Appendix F.6, we provide insights into how the values of these algorithmic hyperparameters impact the performance of RobustTSF.


[R1] Recurrent neural networks and robust time series prediction

Reviewer comment (Experiments):
The characteristics of the chosen datasets need to be described. Hopefully they cover a large range of variability, relationships between input variables, fraction of dataset that have anomalies, etc.

Author response:
**A:**  We would like to clarify that, for the context of learning with noisy labels in time series forecasting with anomalies, datasets are typically selected through two main approaches. One involves manually introducing noise (anomalies) with defined noise ratios into a clean dataset, while the other uses datasets that naturally exhibit real-world anomalies.

In our paper, we employed eight datasets, including Electricity, Traffic, ETT-h1, ETT-h2, ETT-m1, Exchange, with manually added anomalies, and NAB, M4 with real-world anomalies.

Many of these datasets, such as Electricity and Traffic, are widely used in popular time series forecasting research. It's worth noting that even in datasets where anomalies are not manually added, they may still exist inherently due to environmental influences on sensors, such as temperature and humidity. Table 2 demonstrates the commendable performance of RobustTSF even in scenarios without manually added anomalies.

In Appendix F.11, we further demonstrate the robust performance of RobustTSF on a dataset with real-world anomalies.

We provide a comprehensive characterization of these datasets as follows:

|     Dataset      |  LEN  |  FREQ   | Manullay adding anomalies | Train Test ratio |
| :--------------: | :---: | :-----: | :-----------------------: | :--------------: |
|   Electricity    | 26304 |   1h    |            Yes            |       7:3        |
|     Traffic      | 17544 |   1h    |            Yes            |       7:3        |
|      ETT-h1      | 17420 |   1h    |            Yes            |       7:3        |
|      ETT-h2      | 17420 |   1h    |            Yes            |       7:3        |
|      ETT-m1      | 69680 | 15 min  |            Yes            |       7:3        |
|     Exchange     | 7588  |  1 day  |            Yes            |       7:3        |
| NAB (real-cloud) | 4032  | 15 min  |            No             |       7:3        |
|    M4-monthly    | 1871  | Monthly |            No             |       7:3        |

Reviewer comment (Writing):
"Proposation" -> "Proposition."

Author response:
**A:** Thank you for pointing that out; we have corrected it accordingly.

Reviewer comment (Theory):
In section 3, the fourth line, I suspect that $T-\alpha$ should be $T/\alpha$. Is that correct?

Author response:
**A:** It should be $T - \alpha$ since the time series is partitioned into a set of time windows with size $\alpha$, and the total length of the time series is $T$.

Reviewer comment (Presentation):
[Yoon 2022a] and [Yoon 2022b] are the same citation.

Author response:
**A:** Thanks for bringing this to our attention. We have revised and consolidated the citation into one.

Reviewer comment (Presentation):
In appendix B, the lines after the first one seem to have been cut off and are hidden behind figure 2.

Author response:
**A:** Thanks for the careful examination. We have addressed and rectified this issue.