Paper: Towards Dynamic Trend Filtering through Trend Points Detection with Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The authors only claim that ‘DNNs employed by RL agent is capable of capturing intricate patterns within time series data and identifying DTPs based on their non-linearity inherent.’ How do DNNs in RL identify DTPs? Why do you think DNNs in RL can do that? What is the motivation behind that? There are no reasonable discussions on this, despite it being the main contribution of this work.

Author response:
1. How do DNNs in RL can identify DTPs? Why do you think DNNs in RL can do that?
    1. As stated in Section 1, **when using DNN, it is unconstrained by fixed window sizes or frequencies within the time series data domain**. Additionally, deep networks learn in a **data-dependent manner**, enabling them to **capture the characteristics of sub-sequences in time series data more effectively**. This capability enables them to **perform different smoothness on abrupt changes that are distributed differently for each sub-sequence**. For example, as shown in **Figure 3 in a revised version** of the manuscript, DTF-net extracts trends with varying smoothness for each sub-sequence, in contrast to $\ell_1$. Consequently, DTF-net identified and filtered what was considered noise in the third point, while recognizing the other five points as abrupt changes.

Reviewer comment (Presentation):
It is unclear why the authors highlight DTF-net in Table 1 rather than the best performance.

Author response:
3. Why do the authors highlight DTF-net in Table 1 rather than the best performance? (Wavelet overfitting)
    1. In the case of **Wavelet**, it demonstrates **superior performance in terms of simple MSE or MAE metrics when applied to the ground truth of a linear signal added with noise**. However, we conclude that it tends to overfit to noisy data for two reasons. First, **when compared with the experimental results using only a linear signal as the ground truth,** as shown in Table 1 in the revised version, **there is a significant discrepancy with the result of the linear signal added noise as the ground truth**, implying substantial overfitting. Next, as an additional indication of potential overfitting, **we introduce another crucial property of trend filtering, namely, smoothness**. To evaluate this smoothness, **we make the assumption that the added Gaussian noise, accounting for 10%, should be filtered out to consider it an appropriate level of smoothness**. Therefore, Wavelet can be considered incapable of performing effective noise filtering.
    2. We **added the assumption of smoothness, “filtering out at least 10% of Gaussian noise”,** in a revised version and provided updated results in **Table 1**.

Reviewer comment (Experiments):
The authors claim that “Note that wavelets have an overfitting issue, resulting in insufficient noise filtering.” Have you evaluated the noise filtering? How can you show that?

Author response:
3. Why do the authors highlight DTF-net in Table 1 rather than the best performance? (Wavelet overfitting)
    1. In the case of **Wavelet**, it demonstrates **superior performance in terms of simple MSE or MAE metrics when applied to the ground truth of a linear signal added with noise**. However, we conclude that it tends to overfit to noisy data for two reasons. First, **when compared with the experimental results using only a linear signal as the ground truth,** as shown in Table 1 in the revised version, **there is a significant discrepancy with the result of the linear signal added noise as the ground truth**, implying substantial overfitting. Next, as an additional indication of potential overfitting, **we introduce another crucial property of trend filtering, namely, smoothness**. To evaluate this smoothness, **we make the assumption that the added Gaussian noise, accounting for 10%, should be filtered out to consider it an appropriate level of smoothness**. Therefore, Wavelet can be considered incapable of performing effective noise filtering.
    2. We **added the assumption of smoothness, “filtering out at least 10% of Gaussian noise”,** in a revised version and provided updated results in **Table 1**.

Reviewer comment (Experiments):
There are a lot of real-world time-series data available, but the authors did not evaluate the methods on them.

Author response:
4. There are a lot of real-world time-series data. Why did the authors not evaluate the methods on them?
    1. Analyzing trend filtering methods poses two challenges. First, **defining a ground truth for the trend is challenging**. For example, in the case of $\ell_1$, the degree of smoothness varies with $\lambda$, resulting in different trends. Second, **labeling abrupt changes is challenging**. Obtaining real-world data with such labels is difficult, and arbitrary assignment to real-world data is subjective. **Therefore, we performed performance evaluations by setting time series forecasting** as a downstream task, indirectly allowing the assessment of trend filtering results on real-world data.
    2. We **supplemented why trend filtering analysis on real-world data poses challenges** in a revised version.

Reviewer comment (Evaluation):
In Section 3.1.2, the authors fail to provide a satisfactory explanation or analysis for the superiority of random sampling over traditional sequential sampling.

Author response:
1. Section 3.1) Explanation or analysis for the superiority of random sampling over traditional sequential sampling (also aligned with Q1), and is dynamic-length sampling necessary or is equal-length sampling would suffice? (also aligned with Q1)
    1. First, we introduce the revised version of Appendix C.1.
    
    > We conduct an ablation study based on the state encoding method. First, the **construction of episodes is divided into two approaches: a non-sequential method based on random sampling and a sequential method following the conventional time axis order**. Next, the **composition of episode length is categorized into a dynamic approach with random sampling and a static approach using a fixed window**. Note that static length is set to 1500 for Exchange and ETTh1, and 200 for Illness. As shown in Table 5, it is evident that the proposed method, **non-sequential episodes with dynamic length, exhibits the most superior and robust performance**, overall. **Following this, the traditional approach, sequential episodes with static length, shows second robust performance**. This confirms that DTF-net achieves excellent performance through the bidirectional learning it aims for, while the sequential episode with static length method occasionally exhibits overfitting results.
    > 
    
    2. We observed that DTF-net, with its non-sequential episode progression and dynamic length, **achieves robust performance through bidirectional learning** compared to sequential learning with static length. Additionally, it **effectively mitigates overfitting issues** through the sampling approach.
    
    3. We supplemented the detailed ablation study and its results in the revised version of  Appendix C.1.

Reviewer comment (Writing):
In Section 3.1.2, clarity is needed on how the authors determine different lengths for various sampling samples.

Author response:
2. Section 3.1) Clarity is needed on how they determine different lengths for various sampling methods.
    1. We set the starting point and length of the episode's sub-sequence through random sampling from a discrete uniform distribution, as follows,
        
        $$s \sim \text{unif}(0, N),$$
        $$l \sim \text{unif}(h+p, H).$$
        
    2. The reason for using a discrete uniform distribution is to ensure that all sequences are considered uniformly.
    3. We supplemented additional detailed explanations with formulas for random sampling **in a revised version of Section 3.**

Reviewer comment (Novelty):
In Section 3.1.2, the claim that the proposed Positional Encoding is "specifically designed for time series data" appears overstated, as it resembles the positional encoding used in the vanilla Transformer and its variants seen in other TSF Transformers like Informer or Autoformer.

Author response:
3. Section 3.1) Positional Encoding is "specifically designed for time series data" appears overstated.
    1. We acknowledge that the positional encoding statement is overstated and revised it (Section 3.1). Also, we added an **ablation study on positional encoding compared to simple zero padding** methods **in a revised version of Appendix C.1 (Table 5)**.

Reviewer comment (Theory):
In Section 3.2, the choice of MSE loss as the reward function raises questions, as it is unclear why an MSE-based reward ensures effective learning of abrupt changes rather than favoring smoother results, which could lead to lower overall MSE.

Author response:
4. Section 3.2) The reason for the choice of MSE loss at the reward function.
    1. First, we introduce the revised version of Section 3.2.
    
    > In the stock trading task, RL optimizes based on cumulative return as the reward. However, **in general time series, we cannot specify return as a reward.** As a result, similar to traditional trend filtering, **DTF-net aims to preserve the `closeness to the original data' property**, fundamental to trend filtering. However, employing a sum of squared loss function on current values makes it challenging to reflect appropriate smoothness. Therefore, DTF-net distinguishes itself by optimizing for future values instead of current ones. **By employing Time Series Forecasting (TSF) and training RL to minimize its cost function, DTF-net can reflect the characteristics of each sub-sequence and learn temporal dependencies.** This allows adjusting smoothness by tuning the prediction window.
    > 
    
    2. In this context, the **MSE cost function is not solely used but in conjunction with the downstream task of forecasting.** Given that time series forecasting involves learning patterns within time series data, DTF-net is capable of predicting trend points while **taking into account the unique characteristics of sub-sequences**. Additionally, in forecasting tasks, DTF-net predicts trend points while **considering temporal dependencies around abrupt changes (Table 1)**. This is the reason why we chose forecasting MSE cost function as a reward. 
We additionally respond to the second question raised by the reviewer.

The selection of the MSE loss function as a reward is motivated by the $\ell_1$ trend filtering, a widely employed method in time series data. Formally, the $\ell_1$ trend filtering minimizes the following objective function,

$\sum (y_t - x_t)^2 + \lambda \sum |x_{t-1} - 2x_t + x_{t+1}|$.

The first term represents the proximity to the original data, while the second term represents smoothness. Therefore, to capture the proximity property of trend filtering, we utilize the first term as a reward function.

Second, regarding smoothness, $\ell_1$ employs proximity to past-current value and current-future value. 

$|x_{t-1} - 2x_t + x_{t+1}| = |(x_{t-1} - x_t) + (x_{t+1} - x_t)|$

Contrarily, we utilize the proximity of future values based on the prediction, without the additional regulation term, thereby employing forecasting MSE as a reward. We can adjust smoothness by forecasting the window as a hyperparameter as shown in Figure 6. 

We acknowledge the tendency of the forecasting method to lower MSE by generating a smooth output. However, as we leverage the trend extracted through the actions of DTF-net as an additional feature, we take advantage of the characteristic illustrated in Figure 2, where DTF-net performs well in capturing abrupt changes, causing a decrease in MSE. This is further supported by the evidence in Figures 4-(b) and 5, where improved performance in forecasting MSE is observed when the trend effectively reflects abrupt changes.

If you have any further questions or comments regarding the second question, we would be happy to follow up!

Thank you.

Reviewer comment (Writing):
In Section 3.2, while the authors introduce reward sampling as a solution to overfitting, its operation and how it addresses overfitting remain unclear.

Author response:
5. Section 3.2) Reward sampling’s operation and how it addresses overfitting.
    1. First, we introduce the revised version of Section 3.2.
    
    > In here, we found that by using a penalty reward as a negative value, irregularly applying penalties through sampling can prevent overfitting **rather than penalizing at every moment** (Appendix C.2).
    > 
    
    2. Since the reward we use is a penalty with a negative value, RL learns in the direction of reducing the penalty. However, **if the penalty is consistently applied, there is a tendency for the model to overfit to the data**. Therefore, **we intermittently apply the penalty to train DTF-net**, aiming to prevent overfitting.

Reviewer comment (Experiments):
In the Experiment section, an important consideration is whether this method is suitable for multivariate time series forecasting tasks, enabling the detection of abrupt change points across multiple channels simultaneously.

Author response:
6. Experiments) Multivariate time series forecasting
    1. First, we introduce the revised version of the Appendix G.
    
    > DTF-net is applied to multivariate time series data features with the target of univariate trend filtering. This means that RL examines multidimensional aspects to perform trend filtering on the target. However, **individual trend filtering for each dimension in the multidimensional case has not yet been addressed in this study**. Nevertheless, **this is achievable through the multi-discrete action prediction of the PPO algorithm.**
    > 
    
    2. We added multivariate time series forecasting as future work in a revised version of **Appendix G**.

Reviewer comment (Evaluation):
The authors could give a more detailed explanation or analysis of why random sampling is better, and if dynamic-length sampling works better than traditional sequential and equal-length sampling.

Author response:
1. Section 3.1) Explanation or analysis for the superiority of random sampling over traditional sequential sampling (also aligned with Q1), and is dynamic-length sampling necessary or is equal-length sampling would suffice? (also aligned with Q1)
    1. First, we introduce the revised version of Appendix C.1.
    
    > We conduct an ablation study based on the state encoding method. First, the **construction of episodes is divided into two approaches: a non-sequential method based on random sampling and a sequential method following the conventional time axis order**. Next, the **composition of episode length is categorized into a dynamic approach with random sampling and a static approach using a fixed window**. Note that static length is set to 1500 for Exchange and ETTh1, and 200 for Illness. As shown in Table 5, it is evident that the proposed method, **non-sequential episodes with dynamic length, exhibits the most superior and robust performance**, overall. **Following this, the traditional approach, sequential episodes with static length, shows second robust performance**. This confirms that DTF-net achieves excellent performance through the bidirectional learning it aims for, while the sequential episode with static length method occasionally exhibits overfitting results.
    > 
    
    2. We observed that DTF-net, with its non-sequential episode progression and dynamic length, **achieves robust performance through bidirectional learning** compared to sequential learning with static length. Additionally, it **effectively mitigates overfitting issues** through the sampling approach.
    
    3. We supplemented the detailed ablation study and its results in the revised version of  Appendix C.1.

Reviewer comment (Theory):
The authors could explain how reward sampling works in more detail, and why MSE loss as a reward function can help the model effectively learn the information of abrupt changes.

Author response:
5. Section 3.2) Reward sampling’s operation and how it addresses overfitting.
    1. First, we introduce the revised version of Section 3.2.
    
    > In here, we found that by using a penalty reward as a negative value, irregularly applying penalties through sampling can prevent overfitting **rather than penalizing at every moment** (Appendix C.2).
    > 
    
    2. Since the reward we use is a penalty with a negative value, RL learns in the direction of reducing the penalty. However, **if the penalty is consistently applied, there is a tendency for the model to overfit to the data**. Therefore, **we intermittently apply the penalty to train DTF-net**, aiming to prevent overfitting.
4. Section 3.2) The reason for the choice of MSE loss at the reward function.
    1. First, we introduce the revised version of Section 3.2.
    
    > In the stock trading task, RL optimizes based on cumulative return as the reward. However, **in general time series, we cannot specify return as a reward.** As a result, similar to traditional trend filtering, **DTF-net aims to preserve the `closeness to the original data' property**, fundamental to trend filtering. However, employing a sum of squared loss function on current values makes it challenging to reflect appropriate smoothness. Therefore, DTF-net distinguishes itself by optimizing for future values instead of current ones. **By employing Time Series Forecasting (TSF) and training RL to minimize its cost function, DTF-net can reflect the characteristics of each sub-sequence and learn temporal dependencies.** This allows adjusting smoothness by tuning the prediction window.
    > 
    
    2. In this context, the **MSE cost function is not solely used but in conjunction with the downstream task of forecasting.** Given that time series forecasting involves learning patterns within time series data, DTF-net is capable of predicting trend points while **taking into account the unique characteristics of sub-sequences**. Additionally, in forecasting tasks, DTF-net predicts trend points while **considering temporal dependencies around abrupt changes (Table 1)**. This is the reason why we chose forecasting MSE cost function as a reward. 
We additionally respond to the second question raised by the reviewer.

The selection of the MSE loss function as a reward is motivated by the $\ell_1$ trend filtering, a widely employed method in time series data. Formally, the $\ell_1$ trend filtering minimizes the following objective function,

$\sum (y_t - x_t)^2 + \lambda \sum |x_{t-1} - 2x_t + x_{t+1}|$.

The first term represents the proximity to the original data, while the second term represents smoothness. Therefore, to capture the proximity property of trend filtering, we utilize the first term as a reward function.

Second, regarding smoothness, $\ell_1$ employs proximity to past-current value and current-future value. 

$|x_{t-1} - 2x_t + x_{t+1}| = |(x_{t-1} - x_t) + (x_{t+1} - x_t)|$

Contrarily, we utilize the proximity of future values based on the prediction, without the additional regulation term, thereby employing forecasting MSE as a reward. We can adjust smoothness by forecasting the window as a hyperparameter as shown in Figure 6. 

We acknowledge the tendency of the forecasting method to lower MSE by generating a smooth output. However, as we leverage the trend extracted through the actions of DTF-net as an additional feature, we take advantage of the characteristic illustrated in Figure 2, where DTF-net performs well in capturing abrupt changes, causing a decrease in MSE. This is further supported by the evidence in Figures 4-(b) and 5, where improved performance in forecasting MSE is observed when the trend effectively reflects abrupt changes.

If you have any further questions or comments regarding the second question, we would be happy to follow up!

Thank you.

Reviewer comment (Experiments):
The authors could conduct more experiments on multivariate time series if possible or state the challenges for that.

Author response:
6. Experiments) Multivariate time series forecasting
    1. First, we introduce the revised version of the Appendix G.
    
    > DTF-net is applied to multivariate time series data features with the target of univariate trend filtering. This means that RL examines multidimensional aspects to perform trend filtering on the target. However, **individual trend filtering for each dimension in the multidimensional case has not yet been addressed in this study**. Nevertheless, **this is achievable through the multi-discrete action prediction of the PPO algorithm.**
    > 
    
    2. We added multivariate time series forecasting as future work in a revised version of **Appendix G**.

Reviewer comment (Experiments):
Training a deep net might be data- and computational-expensive and may face over-fitting problem. Tuning hyper-parameters is also challenging depending on the data and real-world applications.

Author response:
1. W1) Computational expensive, overfitting problem, and tuning hyper-parameters.
    1. We acknowledge that the limitations of DTF-net include data and computational expenses, the risk of overfitting, and the need for hyperparameter tuning. However, we have made efforts to address or have addressed these issues in the following ways.
    2. **Data-expensive**: While deep networks typically require a substantial amount of data, DTF-net demonstrates robust performance even with small synthetic datasets or Illness data, utilizing a simple MLP policy.
    3. **Computational-expensive**: As discussed in the Appendix, we acknowledge the high computational cost of RL. However, we structured episodes through sampling to optimize DTF-net with a minimal number of steps.
    4. **Overfitting**: We address the overfitting issue through reward sampling. Given that the reward we employ is a penalty with a negative value, RL learns to minimize this penalty. However, continuous application of the penalty may lead to overfitting. To mitigate this, we intermittently apply the penalty during the training of DTF-net, aiming to prevent overfitting.
    5. **Hyper-parameter tuning**: To minimize the number of hyperparameters and ensure the robust performance of DTF-net, we have introduced a sampling methodology.
    6. We supplemented these discussions in a revised version of **Appendix D.2**.

Reviewer comment (Novelty):
The heuristic random sampling algorithm can be optimized by considering the environment/context of the time series trend.

Author response:
2. W2) Heuristic random sampling algorithm, and W3) Overall framework might be built without RL.
    1. Thank you for your suggestions. Specifically, we found methods that solve MDP without RL, including, Dynamic Programming, Linear Programming, Policy Gradient Methods, Monte Carlo Methods, Q-Learning (Model-Free), and Temporal Difference. We plan to add these methods in Appendix G, Future Work section. If the methods that the reviewer wants to specify are not added, we want to hear from you through discussion.

Reviewer comment (Presentation):
Figure 4 is not straightforward.

Author response:
3. Minors 
    2. M2) Figure 4 is not straightforward.
        1. Thank you for bringing these to our attention. We addressed these minor points in the revised version.

Reviewer comment (Presentation):
Legend of Figure 5 is too small, unreadable.

Author response:
3. Minors 
    3. M3) Legend of Figure 5 is too small, unreadable.
        1. Thank you for bringing these to our attention. We addressed these minor points in the revised version.

Reviewer comment (Presentation):
For example, you can formalize the reinforcement learning process with diagrams to enrich the details.

Author response:
1. Formalize the reinforcement learning process with diagrams to enrich the details.
    1. Thank you for your suggestion and **we added a diagram in a revised version of Figure 1.**

Reviewer comment (Experiments):
I noticed that the baselines of several recent sotas compared in your experiment are all geared towards trend filtering, and I'm wondering if their methods take abrupt changes detection into account.

Author response:
2. Compared trend filtering methods take abrupt change detection into account.
    1. **We compared DTF-net with both methods that consider and do not consider abrupt changes.** First, the $\ell_1$ and H-P methods consider abrupt changes using the sum of squared functions. Second, Wavelet is grounded in frequency, and EMD and Median rely on decomposition, which do not prioritize the consideration of abrupt changes. Finally, we also compared with CPD and anomaly detection algorithms, which consider abrupt changes.
    2. We supplemented additional detailed explanations and results **in the revised version of Section 2.1 and Table 1** based on the comments.

Reviewer comment (Experiments):
I am also wondering if there are other methods such as anomaly detection that can be used instead of, or to help, produce similar results.

Author response:
3. Anomaly detection can be used instead of. 
    1. Thank you for your suggestion and **we added experiments on anomaly detection** methods [1] **in the revised version of Table 1.**
    2. Additionally, we plan to add details and other various anomaly detection algorithms [1-3] in Appendix F, Discussion section. **If the methods that the reviewer wants to specify are not added, we want to hear from you through discussion.**
    
    [1] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
    
    [2] *Yiyuan Yang, Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. 2023. DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '23). Association for Computing Machinery, New York, NY, USA, 3033–3045. https://doi.org/10.1145/3580305.3599295*
    
    [3] Jiehui Xu, Haixu Wu, Jianmin Wang, & Mingsheng Long (2022). Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. In *International Conference on Learning Representations*.

We extend our gratitude to ****Reviewer Q1zH**** and present additional responses regarding anomaly detection experiments.

1. We introduced experiments on recent anomaly detection algorithms mentioned above [1-3] in Appendix F.3.
2. We identified two challenges for trend filtering in anomaly detection:
    1. The sensitivity of the threshold to detect anomalies, particularly in capturing extreme value points.
    2. Despite finding an optimal threshold for trend filtering, anomaly detection often misses peak points in complex sub-sequences.

We would again like to thank ****Reviewer Q1zH****, and we hope that our changes adequately address your concerns. Please let us know if you have any further questions or comments, and we are very happy to follow up!