Paper: VMFTransformer: An Angle-Preserving and Auto-Scaling Machine for Multi-horizon Probabilistic Forecasting (ICLR-2024)

Reviewer comment (Experiments):
Many time series datasets exhibit a pronounced seasonal component that dominates the signal. Consequently, it may be challenging to surpass simpler methods that effectively estimate this seasonal component. The paper should address the strength of the seasonal component in the considered datasets.

Author response:
We thank the reviewer for offering valuable feedback. We have addressed each of the concerns raised by the reviewer as outlined below.

1. The reviewers rightly point out that many time series exhibit strong seasonal patterns that can dominate the signal. We analyzed the strength of the seasonal component in the electricity, solar, and traffic datasets using autocorrelation analysis. The electricity data shows a strong daily seasonal pattern with autocorrelation remaining above 0.6 at the 24 hour lag. The solar data shows both daily and yearly seasonal patterns due to the underlying physical processes. The traffic data has relatively weak seasonality. However, basic time series methods such as auto.arima and exponential smoothing often cannot effectively model complex real-world time series. ARIMA, exponential smoothing, and Prophet on our used datasets achieve relatively poor performance compared to more flexible machine learning approaches [1, 2]. These methods make strong assumptions about the underlying data generative processes and do not directly capture uncertainties. Our model demonstrates excellent performance across datasets with varying degrees of seasonality. By jointly modeling the sequential dependence, overall scale, and capturing uncertainty, our method provides robust performance without needing an explicit seasonal component model. We provide quantitative metrics showing our model's ability to accurately forecast across a diverse range of conditions.

[1] Lim, Bryan, et al. "Temporal fusion transformers for interpretable multi-horizon time series forecasting." International Journal of Forecasting 37.4 (2021): 1748-1764.

[2] Wang, Y., Smola, A., Maddix, D., Gasthaus, J., Foster, D., & Januschowski, T. (2019, May). Deep factors for forecasting. In International conference on machine learning (pp. 6607-6617). PMLR.

Reviewer comment (Experiments):
The authors have not included simple benchmarks such as auto.arima or exponential smoothing in their comparisons.

Author response:
We thank the reviewer for offering valuable feedback. We have addressed each of the concerns raised by the reviewer as outlined below.

1. The reviewers rightly point out that many time series exhibit strong seasonal patterns that can dominate the signal. We analyzed the strength of the seasonal component in the electricity, solar, and traffic datasets using autocorrelation analysis. The electricity data shows a strong daily seasonal pattern with autocorrelation remaining above 0.6 at the 24 hour lag. The solar data shows both daily and yearly seasonal patterns due to the underlying physical processes. The traffic data has relatively weak seasonality. However, basic time series methods such as auto.arima and exponential smoothing often cannot effectively model complex real-world time series. ARIMA, exponential smoothing, and Prophet on our used datasets achieve relatively poor performance compared to more flexible machine learning approaches [1, 2]. These methods make strong assumptions about the underlying data generative processes and do not directly capture uncertainties. Our model demonstrates excellent performance across datasets with varying degrees of seasonality. By jointly modeling the sequential dependence, overall scale, and capturing uncertainty, our method provides robust performance without needing an explicit seasonal component model. We provide quantitative metrics showing our model's ability to accurately forecast across a diverse range of conditions.

[1] Lim, Bryan, et al. "Temporal fusion transformers for interpretable multi-horizon time series forecasting." International Journal of Forecasting 37.4 (2021): 1748-1764.

[2] Wang, Y., Smola, A., Maddix, D., Gasthaus, J., Foster, D., & Januschowski, T. (2019, May). Deep factors for forecasting. In International conference on machine learning (pp. 6607-6617). PMLR.

Reviewer comment (Evaluation):
Learning curves should be provided to demonstrate the stability of the training procedure.

Author response:
Learning curves (Figure 3) of our model are provided in the updated version to demonstrate the stability of the training procedure.

Reviewer comment (Evaluation):
The proposed method is trained using maximum likelihood estimation. The authors should also provide the negative log-likelihood (NLL) for all density-based methods.

Author response:
We actually used the negative log-likelihood (NLL) in the training stage. As we explained in our paper ' The training object is to maximize Equation 6 with respect to $\Theta$, or equivalently, minimize the inverse $-l(\Theta)$', where $-l(\Theta)$ refers to the NLL. Thanks for noticing. We now realize this may be a confusing statement and we will clarify it to enhance readability and comprehension.

Reviewer comment (Evaluation):
The authors did not report standard errors and information regarding the number of runs performed.

Author response:
This revised draft incorporates extensive additional experiments in general responses (from part 1 to part 5), now encompassing the latest baseline models (such as PatchTST and DLinear) across varying and longer horizons (24, 168, and 720). Every key result includes the mean and standard deviation to quantify uncertainty.

Reviewer comment (Evaluation):
The results for specific forecast horizons (e.g., h = 1, 2, 3, etc.) should be reported to assess if the procedure increases forecast error for initial horizons while improving the average error across horizons.

Author response:
A comparison of metrics corresponding to different horizons where H ranges from 1 to 720 (Figure 4) of our model is provided in the updated version to demonstrate the stability of forecasting performance.

Reviewer comment (Experiments):
Lack of Recent Benchmarks: The VMFTransformer has not been compared with the most recent benchmarks such as PatchTST [1], or with most comparisons made against older methods. For instance, please consider looking at recent benchmarks provided by https://github.com/timeseriesAI/tsai

Author response:
**W1**, **W2**, and  **W5**: this revised draft incorporates extensive additional experiments in **general responses (from part 1 to part 5)**, now encompassing the latest baseline models (such as PatchTST and DLinear) across varying and longer horizons (24, 168, and 720). Every key result includes the mean and standard deviation to quantify uncertainty.

Reviewer comment (Experiments):
Limited comparison for different horizons: The scope of comparison is limited, which may not adequately reflect the model's performance across a broader range of forecasting scenarios.

Author response:
**W1**, **W2**, and  **W5**: this revised draft incorporates extensive additional experiments in **general responses (from part 1 to part 5)**, now encompassing the latest baseline models (such as PatchTST and DLinear) across varying and longer horizons (24, 168, and 720). Every key result includes the mean and standard deviation to quantify uncertainty.

Reviewer comment (Presentation):
Formatting Issues: There is room for improvement in the formatting of the equations, which could enhance readability and comprehension.

Author response:
**W3**: We will revise the formatting of all equations in the paper to enhance readability and comprehension. Equations are centered on their own lines with consistent spacing around operators. Variable names have been standardized and vector/matrix notation clarified. We believe these changes significantly improve the formatting.

Reviewer comment (Presentation):
Presentation Clarity: The paper could better articulate the main contribution, as the current presentation may be challenging to follow, possibly obscuring the model's innovation.

Author response:
**W4**: The main contribution of our work is the novel probabilistic forecasting model based on the von Mises-Fisher distribution and Transformer architecture. To better articulate this, we will add an explicit contribution statement in the introduction highlighting:
1. The VMFTransformer model itself which captures temporal dependence in a multi-horizon target.
2. The Angle&Scale similarity measure for the self-attention module
3. An efficient method to optimize the likelihood function with the Bessel function

Additionally, we will reorganize several sections to improve logical flow and aid comprehension of the technical details underpinning the model innovation. A graphical abstract will also be added to provide an intuitive overview. We believe these changes address the need to better communicate the main contributions of our work.

Reviewer comment (Experiments):
Insufficient Experimentation: Without more extensive experimentation, it's difficult to ascertain the paper's meaningful contribution to the community and its practical applicability.

Author response:
**W1**, **W2**, and  **W5**: this revised draft incorporates extensive additional experiments in **general responses (from part 1 to part 5)**, now encompassing the latest baseline models (such as PatchTST and DLinear) across varying and longer horizons (24, 168, and 720). Every key result includes the mean and standard deviation to quantify uncertainty.

Reviewer comment (Evaluation):
The current version of the paper solely presents the average value obtained from five trials without including information about the standard deviation. It is highly recommended to include error bars.

Author response:
**W1** and **W4**: Per the reviewers' recommendations, we have added extensive additional experiments with the latest baseline models across longer time horizons. Every key result now includes the mean and standard deviation to rigorously quantify uncertainty. We believe these comprehensive additions showcase our method's strengths.

Reviewer comment (Theory):
Why use the VMF distribution and the truncated normal distribution to characterize the angle and magnitude of the target vector? The motivation behind this is unclear to me.

Author response:
**W2**: You raises an important point regarding the motivation behind using the von Mises-Fisher (VMF) distribution and truncated normal distribution to model the multi-horizon forecasting vector. The key intuition is that the VMF distribution can effectively characterize vectors distributed around a specific direction, which fits our assumption that when dependence exists among the multi-horizon targets, they will concentrate around a certain direction. The truncated normal distribution provides a flexible prior for the scale parameter to make the VMF likelihood tractable. Compared to simply using independent normal or Laplace distributions on each horizon, the VMF and truncated normal distribution can jointly capture the dependence structure via the shared direction parameter in VMF. We will add more discussion on the motivation in the revised paper to clarify this modeling choice.

Reviewer comment (Evaluation):
Metrics used to evaluate uncertainty are not sufficiently convincing, a more commonly used metric, CRPS [1], was not used in the experiment.

Author response:
**W3**: We carefully re-evaluated TransMAF, but unfortunately found it performed poorly on our datasets - significantly worse than the other baseline models we assessed. As such, we decided not to include these inferior results and instead added evaluations using the popular PatchTST and DLinear models, providing readers with a more representative view of the state-of-the-art. Regarding evaluation metrics, we utilize the community standards of MAE, MSE, Q50-Loss, and Q90-Loss - prevalent metrics used across probabilistic time series works.

Reviewer comment (Experiments):
Some probabilistic time series baselines are not compared with the proposed method in the experiment, such as TransMAF [2], [3].

Author response:
**W1** and **W4**: Per the reviewers' recommendations, we have added extensive additional experiments with the latest baseline models across longer time horizons. Every key result now includes the mean and standard deviation to rigorously quantify uncertainty. We believe these comprehensive additions showcase our method's strengths.