Paper: Gaussian Process-Based Corruption-resilience Forecasting Models (ICLR-2024)

Reviewer comment (Theory):
According to Section 2.5, the GP-based corruption model is trained considering the ground truth $Y$ (data to forecast). Does it mean that the proposed joint model can be used only when $Y$ is known? If so, I don't see the point of setting up the problem as a forecast one instead of including $Y$ in the training dataset. If $Y$ is unknown (forecast context), how can the GP parameters be tuned?

Author response:
1. Target variable $Y$ is only known at training time, at test time we are predicting the forecast $Y$, without assuming this knowledge.

Reviewer comment (Evaluation):
Can the authors explain the need to include two metrics (MSE and MAE) in the experimental setups? Since both metrics seek to assess the quality of predictions, the results are redundant. Consequently, half of the tables can be omitted.

Author response:
2. It is common practice to include both MSE and MAE as evaluation metrics in regression tasks. Although, we only included the results for MAE in the Appendix section.

Reviewer comment (Experiments):
Page 8, Section 3.3: Can the authors give further details on the choices (e.g. $\beta_1 = 0.9$, $\beta = 0.98$, and batch size = 256) considered in the numerical implementations?

Author response:
3. The choices of batch size and optimization parameters are based on our preliminary experiments.

Reviewer comment (Experiments):
On page 1, Section 1, the authors suggest that their approach may be adapted to deal with multivariate transient functions but that their focus was on the univariate case. Have they performed numerical examples involving multivariate functions? In any case, can they provide further details on the scalability (in terms of the input dimension) of the framework?

Author response:
4. Univariate forecasting is employed in this context because the datasets featured in the paper are exclusively univariate. In scenarios where the goal is to predict multiple target variables, the objective would be to optimize the MSE loss for multiple targets rather than a single one.

Reviewer comment (Presentation):
A somewhat confusing presentation of the proposed method (see the questions below)

Author response:
The success of our proposed model stems from the division of labors between the first forecasting stage (for coarse-grained) and the denoising stage (fine-grained predictions), where initial forecasts are blurred locally by utilizing a GP model. In between the parameters of the GP model are trained for best end-to-end performance.

Reviewer comment (Experiments):
Lack of examples of the model forecasts apart from the cartoon in Fig. 1

Author response:
To meet your request, we included figures with actual forecasts from our datasets in the paper and the provided link below:

[Actual Forecasts](https://anonymous.4open.science/r/Corruption-resilient-Forecasting-Models-15E8/Actual-forecasts-including-AutoDWC.pdf)

Reviewer comment (Theory):
The reviewer would appreciate further comments and clarification from the authors on why the denoising approach works in this context, noting that providing such clarification could lead to a higher rating.

Author response:
Let us further explain the intuition of our framework. By introducing the GP competent, we envision for a division of responsibilities, where the initial forecasting model predicts the overall (coarse-grained) behavior of the target variable, GP is integrated to introduce local blurring to the initial forecasts. In response, the denoising model focuses on removing the introduced blur by predicting the fine-grained details of the ground-truth. 

The inclusion of the GP component establishes a clear division of tasks between forecasting and denoising. While in the absence of the GP component, the distinction between coarse-grained and fine-grained details becomes less pronounced, potentially causing the denoiser to deprioritize the accurate prediction of fine-grained details. We would also like to emphasize that the parameters of the GP/corruption model are fine-tuned to obtain the best performance.  

We are also open to providing forecast plots of AutoDWC in comparison to our AutoDG(ours) to further illustrate why the division of tasks through GP integration helps to achieve enhanced performance.

Reviewer comment (Experiments):
The isotropic noise baseline (AutoDI) performs similarly to the AutoDWC. Why do you think it is the case? Also, how did you choose the variance of the isotropic noise?

Author response:
The reason stems from the ineffectiveness of removing uncorrelated isotropic noise which is not the typical error mode in SOTA time series forecasting models. Therefore, the inability of AutoDI to outperform AutoDWC is underscored by such limitation. We adjusted the variance of the isotropic noise by fine-tuning the scaled factor $\sigma \in [0, 1]$.

Reviewer comment (Experiments):
Did you try different values of \lambda in Eq. (2)?

Author response:
in our preliminary analysis setting different values for $\lambda$ did not affect the results drastically, however we will include more experiments of adjusting the variable $\lambda$ in Appendix of the camera-ready paper.

Reviewer comment (Presentation):
The presentation of the manuscript can be further improved. For example, in the abstract, I find the mentioning of "using more training data" and "image generation" not necessarily closely related to the point I think the authors were trying to emphasize. Additionally, in Figure 2, $X_1$ and $X_2$ are defined as covariates, which seem to conflict with the previous reference to $X$ as the observations.

Author response:
2. When referring to observations, we include variables, including the target and co-variates, that have been observed in the past. In Figure 2, we provide an example where observations $X$ consist of of $X = \{X_1, X_2, \gamma\}$.

Reviewer comment (Experiments):
I believe the joint framework is quite novel and has clear potential in improving time series forecasting, as demonstrated by the authors very diligently. However, I do wonder whether the authors would be open to compare and evaluate the framework to the calibration model that I mentioned above. My personal belief is they share some commonality between them in how they model the time-correlated noise/signal, but it seems the two objectives functions are still quite different. So I believe it would be interesting to see how they compare both in their model structure, and their performance on a few datasets in practice.

Author response:
1. (& question 1) Upon your request, we included the results of the calibration modeling with GPs denoted as CMGP in the revised version of our paper and at the link below: 

[Adding CMGP](https://anonymous.4open.science/r/Corruption-resilient-Forecasting-Models-15E8/Additional-baselines.pdf)

Reviewer comment (Experiments):
The effectiveness of the proposed framework is marginal when considering the framework doubles the parameter of a single forecasting model, e.g., the denoising model follows the forecasting model's architecture. In this case, it is also worthwhile showing that the benefits of the proposed framework are indeed by its mechanism, instead of overparameterization for better capability with a stack of Informers for example.

Author response:
We include the results of forecasting models with higher number of layers (parameters) in the Appendix and the following link, to show that higher number of parameters does not necessarily result in a better performance due to potential overfitting. 

[Results of standalone forecasting models with higher number of layers (parameters)](https://anonymous.4open.science/r/Corruption-resilient-Forecasting-Models-15E8/Additional-results-higher-number-parameters.pdf)

Reviewer comment (Experiments):
The ablation study is not well-exhibited. Despite there being a simple ablation study for the synthetic data in the Introduction, we expect to see some real-world examples or case studies to prove the statement 'The forecasting model focuses on accurately predicting coarse-grained behavior. The corrupt-denoising model focuses on capturing fine-grained behavior, with a GP model employed to enforce the smoothness and co-relationship in added noise', rather than simply the synthetic data.

Author response:
To meet your request we included the actual forecasts in the paper's Appendix and the provided link. 

[Actual Forecasts](https://anonymous.4open.science/r/Corruption-resilient-Forecasting-Models-15E8/Actual-forecasts-including-AutoDWC.pdf)

Reviewer comment (Evaluation):
Is the reported error, or ground truth normalized?

Author response:
Ground-truth is zero mean normalized, and reported errors are further normalized using the mean of the absolute values of the ground truth data as a scaling factor.

Reviewer comment (Experiments):
Does adding the denoising part to the forecasting model help or harm the convergence?

Author response:
Our forecast-corrupt-denoise framework with GPs contributes to enhanced convergence and showcases a lack of overfitting as the performance on the hold-out test still improves. This is illustrated in the figures provided in the following link and the paper's Appendix section. 

[Convergence during training and validation](https://anonymous.4open.science/r/Corruption-resilient-Forecasting-Models-15E8/Convergence.pdf)