Paper: VQ-TR: Vector Quantized Attention for Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
The manuscript claims that  
>...methods based on Transformers (Vaswani et al., 2017) have dominated the state-of-the-art, outperforming both classical autoregressive approaches, as well as deep learning approaches using Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs).  
However, in e.g. [1], which is not a Transformer based model, results of CRPS are better than the ones reported in the manuscript. A discussion (or comparison) on [1] might help to promote the contribution of the manuscript.

Author response:
In short, the CRPS metric you are most likely referring to in [1] is different from the CRPS metric we are computing in our paper; the former takes smaller values than the latter. This is a common misunderstanding in the literature, and we appreciate the opportunity to clarify.

In more explicit detail, note that [1] is a multivariate model, that learns the full joint distribution of all time series together; the results you most likely refer to are in terms of the CRPS-SUM metric, which is different from the univariate metrics CRPS. Importantly for comparing these works, CRPS-SUM is smaller than the corresponding CRPS metric. The appendix of [1] contains the CRPS table where one observes a recurring pattern in time series research, namely that the univariate models (i.e. all those considered in our paper) typically perform better than their multivariate counterparts for the smallish datasets being used in scientific research. This phenomenon can also be observed in the DLinear or PatchTST and similar papers where even a simple linear model is shown to be better than transformer-based models, although this point of multivariate-vs-univariate is not made clear in those works either.

Reviewer comment (Writing):
In Sec. 4.2, the use of ```\citet{}``` and ```\citep{}``` needs to be carefully handled.

Author response:
Thank you for taking the time to review our paper, for your insightful comments, and for acknowledging the novelty and benefit of our methodology! We will fix the formatting issues you noted regarding citations, and we provide detailed responses to your questions and criticisms below.

Reviewer comment (Experiments):
Can you elaborate the influence of different codebook sizes?

Author response:
Please see our general comments to all reviewers (titled: “Response RE Codebook Sizes”) for a very detailed discussion about the influence of codebook size, and the stability of this effect. In summary, the main takeaway from the paper should be that for the most part performance is fairly robust to codebook size, with changes in mean performance due to varying codebook size that are very small in absolute value. However, we do see evidence that in some cases, especially for datasets where the underlying structure is simpler and strong regularization is needed (i.e. Traffic and Taxi), there is an empirical benefit of using especially small codebook sizes. We will add additional discussions to the final paper to make the influence of changing the codebook size more clear.

Reviewer comment (Experiments):
How is the context window length C selected?

Author response:
See our general response to all reviewers for detail about this (titled: “Response RE Context Window Sizes”). In short, the context window is a hyperparameter that is optimized on a per method/dataset basis.

Reviewer comment (Novelty):
Motivation, exposition and the key idea follow closely from VQ-AR. This is however justified since the computational benefits to the transformer architecture are stronger. However, the contrast could be made clearer in the paper.

Author response:
For our response to your comment about the closeness of our methodology with VQ-AR, and about what context windows were used in experiments, please see our general responses to all reviewers (titled: “Response RE VQ-TR vs VQ-AR” and “Response RE Context Window Sizes”).

Reviewer comment (Experiments):
The choice of the distribution output per dataset remains an open point. How was this choice made for the baselines? The fact that -iqn and -t survive for VQ-TR in wiki and taxi respectively invites the question how these hyperparameters were selected.

Author response:
# Choice of Output Distributions

In order to have a more direct one-to-one comparison versus VQ-AR, for each dataset we used the same kind of distribution head for VQ-TR as the best performing distribution head of VQ-AR. In general, we used -t for datasets with continuous data, and -nb for datasets with count based data, with the exception that we used -iqn for Solar and Wikipedia (because this is what performed best for VQ-AR). In addition, we used different distribution heads for different datasets in order to highlight and reinforce the fact that the output head can be changed within the VQ-TR framework. Of course, one could perform explicit hyperparameter optimization on the choice of distribution head (which we did not do for the above reasons), and if we did this the performance of VQ-TR would potentially be even better. We will clarify this.

In addition, for the Taxi dataset VQ-TR used a negative binomial head (same as VQ-AR), the -t suffix for Taxi is a typo that we will fix for the final paper.

Reviewer comment (Evaluation):
Why does the model have higher runtime than TFT? Despite the comment that similar batch sizes were used, one would expect that the quantized architecture still outperforms TFT.

Author response:
# Why does TFT have a Shorter Running Time

The reason for this is that, unlike the other transformer-based methods, TFT by design only has a single self-attention module. Therefore, hyperparameter changes to increase the complexity/capacity of the model can only go towards layers outside of the attention mechanism, so no matter what hyperparameters are chosen it is always relatively fast compared with the other transformer-based baselines. That being said, it still has quadratic in context length scaling. Note that our comparison in Figure 2 does not show the asymptotic scaling of computational cost as context length increases, rather we show what the computation cost looks like in practice for a reasonably typical context length on this dataset. Also, it should be noted that although TFT is often fast in practice, we find that it performs relatively poorly on many datasets.

Reviewer comment (Experiments):
In the appendix the authors write: "Note that we can afford to use a longer context length of C = 20 × P for VQ-TR due to its memory efficiency, as noted in Figure 2." Was the context length fixed across models or did VQ-TR get a head start?

Author response:
For our response to your comment about the closeness of our methodology with VQ-AR, and about what context windows were used in experiments, please see our general responses to all reviewers (titled: “Response RE VQ-TR vs VQ-AR” and “Response RE Context Window Sizes”).