Paper: Variational quantization for state space models (ICLR-2024)

Reviewer comment (Presentation):
The writing of this paper can be improved. On Page 2, there is a relatively large blank space, which can be optimized.

Author response:
We thank the referee for his appreciation of our work and we apologize for these mistakes. They were corrected and we improved the layout of our paper. Regarding Figure 1, the trajectory of hidden states displayed in the figure is an example of a possible sequence of hidden states. Thus, this trajectory is not fixed but sampled using the laws of hidden states. A sentence has been added to the figure description to clarify this point.

Reviewer comment (Writing):
There are many typos and grammatical issues in this paper.

Author response:
We thank the referee for his appreciation of our work and we apologize for these mistakes. They were corrected and we improved the layout of our paper. Regarding Figure 1, the trajectory of hidden states displayed in the figure is an example of a possible sequence of hidden states. Thus, this trajectory is not fixed but sampled using the laws of hidden states. A sentence has been added to the figure description to clarify this point.

Reviewer comment (Writing):
On page 6, Section 3.1.3, “…where T stand for…”, “stand” should be “stands”.

Author response:
We thank the referee for his appreciation of our work and we apologize for these mistakes. They were corrected and we improved the layout of our paper. Regarding Figure 1, the trajectory of hidden states displayed in the figure is an example of a possible sequence of hidden states. Thus, this trajectory is not fixed but sampled using the laws of hidden states. A sentence has been added to the figure description to clarify this point.

Reviewer comment (Writing):
On Page 8, in the caption of Table 1, “…their associated MASE…”, “MASE” should be “MASEs”.

Author response:
We thank the referee for his appreciation of our work and we apologize for these mistakes. They were corrected and we improved the layout of our paper. Regarding Figure 1, the trajectory of hidden states displayed in the figure is an example of a possible sequence of hidden states. Thus, this trajectory is not fixed but sampled using the laws of hidden states. A sentence has been added to the figure description to clarify this point.

Reviewer comment (Writing):
On Page 8, Section 3.2.2, the second equation should be MAE.

Author response:
We thank the referee for his appreciation of our work and we apologize for these mistakes. They were corrected and we improved the layout of our paper. Regarding Figure 1, the trajectory of hidden states displayed in the figure is an example of a possible sequence of hidden states. Thus, this trajectory is not fixed but sampled using the laws of hidden states. A sentence has been added to the figure description to clarify this point.

Reviewer comment (Reproducibility):
In Figure 1, for Hidden States Trajectory, e.g., $\hat{x}_{t+1}^i=2$, are those “2,2,1,1,3,...,3” fixed or flexible to adjust in your implementation?

Author response:
We thank the referee for his appreciation of our work and we apologize for these mistakes. They were corrected and we improved the layout of our paper. Regarding Figure 1, the trajectory of hidden states displayed in the figure is an example of a possible sequence of hidden states. Thus, this trajectory is not fixed but sampled using the laws of hidden states. A sentence has been added to the figure description to clarify this point.

Reviewer comment (Experiments):
The two-stage training appears unnecessary in the context of time series. It may lead to sub-optimal results.

Author response:
Thank you for this thoughtful question. In this paper we propose an efficient, simple and stable way of training our method by training alternatively the emission laws and the prior law of the hidden states. To motivate the proposed two-steps training, we added a justification part in the main text (Section 2.2). Furthermore, a section in Appendix was also added where we compare a one-step training with the proposing one, pointing out instabilities solved by the two-steps training. We acknowledge that additional fine tuning of the implementation could be proposed to improve the training process. This question remains open and will be tackled in the future as well as research perspectives raised in the conclusion of our work.

Reviewer comment (Experiments):
How well does your method work on long context forecasting problem?

Author response:
In addition to the experiments conducted on the fashion dataset (with a horizon set to 52), we also evaluated our method on a collection of 8 benchmark datasets in Section 3.2. The main motivation for this second set of experiments was to evaluate our approach on different forecasting use cases and evaluate its behavior over different forecasting horizons. We found that the proposed approach can be easily used in a wide range of forecasting tasks while providing accurate forecasts that challenge other state-of-the-art methods.

Reviewer comment (Experiments):
While the emprical results are nice, it would further strengthen the paper if the authors could perform ablation experiments to uncover/provide a better intuition about why their algorithm performs well against others.

Author response:
We thank the referee for this suggestion. Following this remark, several ablation studies were added in the appendix. For instance, in Appendix A.6, we provided more justification on the two-steps training process presented in the paper, showing how it solves some training instabilities and improves final model accuracy. Likewise, we reviewed in Appendix A.7 the impact of learning the variance of the emission distributions and in Appendix A.8, the impact of using a Markov chain for the latent states.

Reviewer comment (Novelty):
The author's new algorithm performs almost similarly to previous SOTA Transformer-based model PatchTST/64. However, it's unclear from my first reading why one would prefer one over the other. Is it easier to train/better runtime etc.?

Author response:
We thank the referee for this thoughtful remark and recognize that the motivation for the method presented was not clearly defined enough in the paper. One argument in favor of our method is that it provides probabilistic predictions that allow users to build confidence intervals. On the Fashion use case, by analysing when and where the emission laws having access to the influencers signals is activated, we could spot specific regime shifts in the time series and understand how the proposed approach succeeded at forecasting these challenging patterns. A natural future work would be to propose more complex generative models where Transformer architectures could be used in the transition densities and emission distributions. Such models would benefit from the state space framework and highlight the usefulness of the two stage procedures.

Reviewer comment (Experiments):
It's nice that the authors performed a grid search over learning rates and batch size for the other algorithms. I think the paper would be further strengthened if the authors conducted a hyperparameter search also over network size where it make sense (similar to the grid search that the authors performed over hidden size for their network)?

Author response:
We thank the referee for this suggestion. Thanks to this comment, we provided a complementary ablation study presented in appendix A.5. In this section, we performed a grid search on the number of parameters and compared the optimal size found for our approach with the size of other state-of-the-art methods. We found that our model performed well with an architecture with 700,000 parameters, which is superior to PatchTST, Informer, and DeepAR methods and inferior to N-HiTS, N-BEATS, and TimesNet methods. Considering recent published models for natural language processing or visual detections, there is still a huge margin regarding the size of all current time series forecasting methods.