Original: VQ-TR : V ECTOR Q UANTIZED A TTENTION FOR T IME S ERIES F ORECASTING
Revised: VQ-TR : V ECTOR Q UANTIZED A TTENTION FOR T IME S ERIES F ORECASTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern time series datasets can easily contain hundreds or thousands of temporal time points, however, Transformer based models scale poorly to the size of the sequence length constraining their context size in the seq-to-seq setting.
Revised: Modern time series datasets can easily contain hundreds or thousands of temporal time points, however, Transformer based models scale poorly to the size of the sequence length constraining their context size in the seq-to-seq setting.

Original: In this work, we introduce VQ-TR which maps large sequences to a discrete set of latent representations as part of the Attention module.
Revised: In this work, we introduce VQ-TR which maps large sequences to a discrete set of latent representations as part of the Attention module.

Original: This allows us to attend over larger context windows with linear complexity with respect to the sequence length.
Revised: This allows us to attend over larger context windows with linear complexity with respect to the sequence length.

Original: We compare this method with other competitive deep learning and classical univariate probabilistic models and highlight its performance using both probabilistic and point forecasting metrics on a variety of open datasets from different domains.
Revised: We compare this method with other competitive deep learning and classical univariate probabilistic models and highlight its performance using both probabilistic and point forecasting metrics on a variety of open datasets from different domains.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning based probabilistic time series forecasting models (Benidis et al., 2022) typically consists of a component that learns representations of a time series’ history, while another component learns some emission model (point or probabilistic) conditioned on this representation of the history.
Revised: Deep learning based probabilistic time series forecasting models (Benidis et al., 2022) typically consists of a component that learns representations of a time series’ history, while another component learns some emission model (point or probabilistic) conditioned on this representation of the history.

Original: One can typically use Recurrent Neural Networks (RNNs), Casual Convolutional networks or the popular architecture at time of writing namely, Transformers (Vaswani et al., 2017) to learn historical representations.
Revised: One can typically use Recurrent Neural Networks (RNNs), Casual Convolutional networks or the popular architecture at time of writing namely, Transformers (Vaswani et al., 2017) to learn historical representations.

Original: Transformers offer good inductive bias for the forecasting task, as they can look back over the full context history of a time series, while RNNs suffer from forgetting and convolutions network have limited temporal receptive fields for the amount of parameters.
Revised: Transformers offer good inductive bias for the forecasting task (Zhou et al., 2021), as they can look back over the full context history of a time series, while RNNs suffer from forgetting and convolutions network have limited temporal receptive fields for the amount of parameters.

Original: Transformers however suffer from quadratic complexity for memory and compute in the size of sequence over which they are learning representations.
Revised: Transformers however suffer from quadratic complexity for memory and compute in the size of sequence over which they are learning representations.

Original: This constrains the size of contexts over which Transformer based models can make predictions over which can potentially hinder these models from learning long-range dependencies as well as constraining the depth of these models leading to poorer representations being learnt.
Revised: This constrains the size of contexts over which Transformer based models can make predictions over which can potentially hinder these models from learning long-range dependencies as well as constraining the depth of these models leading to poorer representations being learnt.

Original: A number of techniques have been developed to deal with this issue by reducing the computation or reducing the sequence length.
Revised: A number of techniques have been developed to deal with this issue by reducing the computation or reducing the sequence length.

Original: In this work we start with an observation on the approximation of the Attention module in the
Revised: In this work we start with an observation on the approximation of the Attention module in the

Original: Transformer and design a novel encoder-decoder Transformer based architecture for the forecasting use case which is linear in its computational and memory use with respect to the sequence size.
Revised: Transformer and design a novel encoder-decoder Transformer based architecture for the forecasting use case which is linear in its computational and memory use with respect to the sequence size.

Original: We do this by incorporating a Vector Quantization (van den Oord et al., 2017) into the Transformer which allows us to learn discrete representations in a hierarchical fashion.
Revised: We do this by incorporating a Vector Quantization (van den Oord et al., 2017) into the Transformer which allows us to learn discrete representations in a hierarchical fashion.

Original:  B ACKGROUND 2. P ROBABILISTIC T IME S ERIES F ORECASTING
Revised:  B ACKGROUND 2. P ROBABILISTIC T IME S ERIES F ORECASTING

Original: The task of probabilistic time series forecasting in the univariate setting consists of training on a dataset of D ≥ 1 time series D train = { x i 1: T i } where i ∈ { 1 , . . .
Revised: The task of probabilistic time series forecasting in the univariate setting consists of training on a dataset of D ≥ 1 time series D train = { x i 1: T i } where i ∈ { 1 , . . .

Original: , D } and at each time point t we have x it ∈ R or in N .
Revised: , D } and at each time point t we have x it ∈ R or in N .

Original: We are tasked with predicting the potentially complex distribution of the next
Revised: We are tasked with predicting the potentially complex distribution of the next

Original: P > 1 time steps into the future and we are given a back-testing test set D test =
Revised: P > 1 time steps into the future and we are given a back-testing test set D test =

Original: { x i T i +

Revised: { x i T i +

Original: : T i + P } .
Revised: : T i + P } .

Original: Each time index t is in reality a date-time value which increments regularly based on the frequency of the dataset in question and the last training point T i for each time series may or may not be the same date-time.
Revised: Each time index t is in reality a date-time value which increments regularly based on the frequency of the dataset in question and the last training point T i for each time series may or may not be the same date-time.

Original: Autoregressive models like (Graves, 2013; Salinas et al., 2019b) estimate the prediction density by decomposing the joint distribution of all P points via the chain-rule of probability as: p X ( x iT i +1:
Revised: Autoregressive models like (Graves, 2013; Salinas et al., 2019b) estimate the prediction density by decomposing the joint distribution of all P points via the chain-rule of probability as: [Equation] [Equation] P

Original: T i + P ) ≈ Π P t =1 p ( x iT i + t | x i 1: T i − 1+ t , c i 1: T i + P ; θ ) , parameterized by some model with trained weights θ .
Revised: [Equation] θ ) [Equation] parameterized by some model with trained weights θ .

Original: This requires that the next time point is conditioned on all the past and covariates
Revised: This requires that the next time point is conditioned on all the past and covariates

Original: c it (detailed in Section 3.3), which is computationally challenging to scale especially if the time series has a large history.
Revised: c it (detailed in Section 3.3), which is computationally challenging to scale especially if the time series has a large history.

Original: Models like DeepAR (Salinas et al., 2019b) typically resort to the seq-to-seq paradigm (Sutskever et al., 2014) and consider some context window of fixed sized C sampled randomly from the full time series history to learn some historical representation and use this representation in the decoder to learn the distribution of the subsequent time points of the context.
Revised: Models like DeepAR (Salinas et al., 2019b) typically resort to the seq-to-seq paradigm (Sutskever et al., 2014) and consider some context window of fixed sized C sampled randomly from the full time series history to learn some historical representation and use this representation in the decoder to learn the distribution of the subsequent time points of the context.

Original: This does however mean that the model falls short of capturing long-range temporal dependencies in its prediction which can lead to worse approximation of the future distribution.
Revised: This does however mean that the model falls short of capturing long-range temporal dependencies in its prediction which can lead to worse approximation of the future distribution.

Original: Encoder-decoder Transformers (Vaswani et al., 2017) naturally fit the seq-to-seq paradigm where N encoding Transformer layers can be used to learn a context window sized sequence of representations denoted by:
Revised: Encoder-decoder Transformers (Vaswani et al., 2017) naturally fit the seq-to-seq paradigm where N encoding Transformer layers can be used to learn a context window sized sequence of representations denoted by:

Original: [Equation]
Revised: [Equation]

Original: Afterwards M layers of a causal or masked decoding Transformer can be used to model the subsequent P future time points conditioned on the encoding representations as:
Revised: Afterwards M layers of a causal or masked decoding Transformer can be used to model the subsequent P future time points conditioned on the encoding representations as:

Original: [Equation]
Revised: [Equation]

Original: For example if we assume the data comes from a Gaussian then the outputs from the Transformer’s
Revised: For example if we assume the data comes from a Gaussian then the outputs from the Transformer’s

Original: M decoders can be passed to a layer which returns appropriately signed parameters of a Gaussian whose log-likelihood, given by
Revised: M decoders can be passed to a layer which returns appropriately signed parameters of a Gaussian whose log-likelihood, given by

Original: [Equation]
Revised: [Equation]

Original: can be maximized for all i and t from the D train using stochastic gradient descent (SGD) as detailed in Section 3.1.
Revised: can be maximized for all i and t from the D train using stochastic gradient descent (SGD) as detailed in Section 3.1.

Original: Although Transformers offer a better alternative to recurrent neural networks (RNN) (like the LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Chung et al., 2014) which apart from being sequential suffer from forgetting with large context windows) or Convolutional models like TCN (Bai et al., 2018) (which have limited temporal receptive fields) they scale, in terms of compute and memory, quadratically with the size of sequence length per layer.
Revised: Although Transformers offer a better alternative to recurrent neural networks (RNN) (like the LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Chung et al., 2014) which apart from being sequential suffer from forgetting with large context windows) or Convolutional models like TCN (Bai et al., 2018) (which have limited temporal receptive fields) they scale, in terms of compute and memory, quadratically with the size of sequence length per layer.

Original: To reduce the computational requirements of Transformers, which is an active area of research, one can employ a number of strategies, for example by compressing the sequence, exploiting locality or by mitigating computation for each of the input entity.
Revised: To reduce the computational requirements of Transformers, which is an active area of research, one can employ a number of strategies, for example by compressing the sequence, exploiting locality or by mitigating computation for each of the input entity.

Original: 2. V ECTOR Q UANTIZATION (VQ)
Revised: 2. V ECTOR Q UANTIZATION (VQ)

Original: The VQ-VAE (van den Oord et al., 2017; Razavi et al., 2019) is an encoder-decoder Variational
Revised: The VQ-VAE (van den Oord et al., 2017; Razavi et al., 2019) is an encoder-decoder Variational

Original: Autoencoder (VAE) (Kingma & Welling, 2019) that maps inputs onto a set of J ≥ 1 discrete latent variables called the codebook { z 1 , . .
Revised: Autoencoder (VAE) (Kingma & Welling, 2019) that maps inputs onto a set of J ≥ 1 discrete latent variables called the codebook { z 1 , . .

Original: z J } , and a decoder that reconstructs the inputs from the resulting discrete vectors.
Revised: z J } , and a decoder that reconstructs the inputs from the resulting discrete vectors.

Original: The input vector is quantized with respect to its distance to its nearest codebook vector:
Revised: The input vector is quantized with respect to its distance to its nearest codebook vector:

Original: [Equation]
Revised: [Equation]

Original: The codebook is learned by back-propgation of the gradient coming upstream of the VQ module and due to the non-differentiable operation one uses the Straight-Through gradient estimator (Hinton et al., 2012; Bengio et al., 2013) to copy the gradients downstream.
Revised: The codebook is learned by back-propgation of the gradient coming upstream of the VQ module and due to the non-differentiable operation one uses the Straight-Through gradient estimator (Hinton et al., 2012; Bengio et al., 2013) to copy the gradients downstream.

Original: Additionally the VQ has two extra losses namely the latent loss which encourages the alignment of the codebook vectors to the inputs of the VQ as well as a commitment loss which penalizes the inputs from switching codebook vectors too frequently.
Revised: Additionally the VQ has two extra losses namely the latent loss which encourages the alignment of the codebook vectors to the inputs of the VQ as well as a commitment loss which penalizes the inputs from switching codebook vectors too frequently.

Original: This is done via the “stop-gradient” or “detach” operators of deep learning frameworks which blocks gradients from flowing into its argument.
Revised: This is done via the “stop-gradient” or “detach” operators of deep learning frameworks which blocks gradients from flowing into its argument.

Original: Thus the additional two VQ losses can be written as:
Revised: Thus the additional two VQ losses can be written as:

Original: [Equation]
Revised: [Equation]

Original: where β is the hyperparameter weighting the commitment loss.
Revised: where β is the hyperparameter weighting the commitment loss.

Original: Since the optimal codes would be the k-means clusters of the input representations, van den Oord et al.
Revised: Since the optimal codes would be the k-means clusters of the input representations, van den Oord et al.

Original: (2017) provides an exponential moving average training scheme of the latents instead of the latent loss (first term of (2)).
Revised: (2017) provides an exponential moving average training scheme of the latents instead of the latent loss (first term of (2)).

Original: Additionally, to aid learning, Jukebox (Dhariwal et al., 2020) proposes to replace the codebook vectors that have an exponential moving average cluster size less than some threshold by a random incoming vector from the batch.
Revised: Additionally, to aid learning, Jukebox (Dhariwal et al., 2020) proposes to replace the codebook vectors that have an exponential moving average cluster size less than some threshold by a random incoming vector from the batch.

Original:  VQ-TR M ODEL
Revised:  VQ-TR M ODEL

Original: We motivate this method with an observation on the effect of approximations of the query vector in self-attention.
Revised: We motivate this method with an observation on the effect of approximations of the query vector in self-attention.

Original: Recall that in self-attention the incoming sequence of vectors are mapped to query, key, and value vectors.
Revised: Recall that in self-attention the incoming sequence of vectors are mapped to query, key, and value vectors.

Original: For each t indexed vector, this is denoted by q t , k t , and v t , respectively.
Revised: For each t indexed vector, this is denoted by q t , k t , and v t , respectively.

Original: Let us denote the approximation of the query vector q t by ˆ q t .
Revised: Let us denote the approximation of the query vector q t by ˆ q t .

Original: The attention weight for step t attending on some u step is (Phuong & Hutter, 2022)
Revised: The attention weight for step t attending on some u step is (Phuong & Hutter, 2022)

Original: [Equation]
Revised: [Equation]

Original: and the output representation is denoted by o t , where o t = (cid:80) u w tu v u .
Revised: and the output representation is denoted by o t , where o t = (cid:80) u w tu v u .

Original: We then have the following:
Revised: We then have the following:

Original: Theorem 1.
Revised: Theorem 1.

Original: If max (cid:12) (cid:12) q T t k u − ˆ q Tt k u (cid:12) (cid:12) ≤ δ then for sufficiently small δ > 0 the attention weight with respect to the approximation ˆ q t given by ˆ w tu is bounded by
Revised: If max (cid:12) (cid:12) q T t k u − ˆ q Tt k u (cid:12) (cid:12) ≤ δ then for sufficiently small δ > 0 the attention weight with respect to the approximation ˆ q t given by ˆ w tu is bounded by

Original: [Equation]
Revised: [Equation]

Original: and as a result the output representation is | o
Revised: and as a result the output representation is | o

Original: [Equation]
Revised: [Equation]

Original: − ˆ o t | ⪯ 2 δ o t .
Revised: − ˆ o t | ⪯ 2 δ o t .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Since, max j (cid:12)(cid:12) q
Revised: Since, max j (cid:12)(cid:12) q

Original: Tt k j − ˆ q Tt k j (cid:12)(cid:12) ≤ δ , then exp( − δ ) ≤ exp
Revised: Tt k j − ˆ q Tt k j (cid:12)(cid:12) ≤ δ , then exp( − δ ) ≤ exp

Original: [Equation]
Revised: [Equation]

Original: q Tt k j − ˆ
Revised: q Tt k j − ˆ

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: assuming δ is small,
Revised: assuming δ is small,

Original: [Equation]
Revised: [Equation]

Original: or,
Revised: or,

Original: [Equation]
Revised: [Equation]

Original: Since o t = (cid:80) u w tu v u ,
Revised: Since o t = (cid:80) u w tu v u ,

Original: [Equation]
Revised: [Equation]

Original: With the above result we see that we can bound the error in the output representation of self-attention as a result of approximating the query vector.
Revised: With the above result we see that we can bound the error in the output representation of self-attention as a result of approximating the query vector.

Original: We can see how to make sure δ is small using a discrete set of approximations given by { z 1 , . . .
Revised: We can see how to make sure δ is small using a discrete set of approximations given by { z 1 , . . .

Original: , z J } to the query vectors.
Revised: , z J } to the query vectors.

Original: We ideally want to
Revised: We ideally want to

Original: [Equation]
Revised: [Equation]

Original: or,
Revised: or,

Original: [Equation]
Revised: [Equation]

Original: Instead of minimizing the maximum over all possible q , k , we can minimize the sum or mean, i.e.,
Revised: Instead of minimizing the maximum over all possible q , k , we can minimize the sum or mean, i.e.,

Original: [Equation]
Revised: [Equation]

Original: z 1 ,...,
Revised: z 1 ,...,

Original: [Equation]
Revised: [Equation]

Original: Letting ∥ x ∥ 2 M = x T Mx
Revised: Letting ∥ x ∥ 2 M = x T Mx

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is same as the weighted K-means objective.
Revised: which is same as the weighted K-means objective.

Original: The weights depend on the covariance Σ k and mean µ k of the key vectors.
Revised: The weights depend on the covariance Σ k and mean µ k of the key vectors.

Original: Thus, in order to learn good K-means approximations of the query vectors from a discrete set, we introduce the following model.
Revised: Thus, in order to learn good K-means approximations of the query vectors from a discrete set, we introduce the following model.

Original: In the VQ-TR model we modify the Transformer’s encoder architecture by first mapping the C incoming vectors, denoted by X ∈ R C × F , through a VQ module:
Revised: In the VQ-TR model we modify the Transformer’s encoder architecture by first mapping the C incoming vectors, denoted by X ∈ R C × F , through a VQ module:

Original: [Equation]
Revised: [Equation]

Original: which will return the sequence of C indices of the set of only J vectors denoted by Z 0 ∈ R J × F .
Revised: which will return the sequence of C indices of the set of only J vectors denoted by Z 0 ∈ R J × F .

Original: We can apply Transformer based cross-attention to obtain latent Z 1 ∈ R J × F :
Revised: We can apply Transformer based cross-attention to obtain latent Z 1 ∈ R J × F :

Original: [Equation]
Revised: [Equation]

Original: Since there are only J latent vectors and typically in practice J ≪ C , we can process them further via self-attention L times:
Revised: Since there are only J latent vectors and typically in practice J ≪ C , we can process them further via self-attention L times:

Original: [Equation]
Revised: [Equation]

Original: Finally, we return the original number of sequence by gathering the resulting latent via the indices with respect to the quantization of the input vectors X :
Revised: Finally, we return the original number of sequence by gathering the resulting latent via the indices with respect to the quantization of the input vectors X :

Original: [Equation]
Revised: [Equation]

Original: This construction leads to an architecture with memory and compute complexity of O ( CJ ) +
Revised: This construction leads to an architecture with memory and compute complexity of O ( CJ ) +

Original: O ( LJ 2 ) from the the cross-attention and latent self-attention respectively (Jaegle et al., 2021;
Revised: O ( LJ 2 ) from the the cross-attention and latent self-attention respectively (Jaegle et al., 2021;

Original: Hawthorne et al., 2022) per number of encoding layers N .
Revised: Hawthorne et al., 2022) per number of encoding layers N .

Original: One downside however to this architecture is that we lose the ability to model causal latents.
Revised: One downside however to this architecture is that we lose the ability to model causal latents.

Original: On the other hand, as is commonly done in time series forecasting, we will use this non-causal encoder part of our architecture to learn discrete representations of large context windows, while the decoder will be the causal Transformer decoder and therefore scale O ( MP 2 ) for M decoding layers.
Revised: On the other hand, as is commonly done in time series forecasting, we will use this non-causal encoder part of our architecture to learn discrete representations of large context windows, while the decoder will be the causal Transformer decoder and therefore scale O ( MP 2 ) for M decoding layers.

Original: Since P ≪ C for the datasets we train on, this will not hinder us from training or doing inference conditioned on large histories.
Revised: Since P ≪ C for the datasets we train on, this will not hinder us from training or doing inference conditioned on large histories.

Original: We present a schematic of the VQ-TR model in Figure 1 for both the training (Section 3.1) and inference (Section 3.2) scenarios.
Revised: We present a schematic of the VQ-TR model in Figure 1 for both the training (Section 3.1) and inference (Section 3.2) scenarios.

Original: A added benefit of this approach is that we can mitigate the quadratic memory and computation issue caused by a large sequence of input vectors in the vanilla Transformer to learn long term dependencies.
Revised: A added benefit of this approach is that we can mitigate the quadratic memory and computation issue caused by a large sequence of input vectors in the vanilla Transformer to learn long term dependencies.

Original: [Figure]
Revised: [Figure]

Original: 3. T RAINING
Revised: 3. T RAINING

Original: Given a set D train of D ≥ 1 time series, we construct batches B of inputs by randomly sampling time series { x i 1: T i } , with i ∈ Z + such that i ≤ D , then selecting random t ∈ Z + with t ≤ T i − C − P , and sampling context windows { x it : t + C } , and subsequent prediction windows { x it + C : t + C + P } , for fixed context window length C and prediction window length P .
Revised: Given a set D train of D ≥ 1 time series, we construct batches B of inputs by randomly sampling time series { x i 1: T i } , with i ∈ Z + such that i ≤ D , then selecting random t ∈ Z + with t ≤ T i − C − P , and sampling context windows { x it : t + C } , and subsequent prediction windows { x it + C : t + C + P } , for fixed context window length C and prediction window length P .

Original: We can then for each batch step minimize the negative log-likelihood of the predicted distribution with respect to the ground truth predictions together with the N latent and commitment losses from the VQ module of the encoder jointly.
Revised: We can then for each batch step minimize the negative log-likelihood of the predicted distribution with respect to the ground truth predictions together with the N latent and commitment losses from the VQ module of the encoder jointly.

Original: This is in contrast to the practice of first learning the discrete latent representations in an unsupervised fashion and then using these latents for down stream tasks as in for example the DALL·E (Ramesh et al., 2021) model.
Revised: This is in contrast to the practice of first learning the discrete latent representations in an unsupervised fashion and then using these latents for down stream tasks as in for example the DALL·E (Ramesh et al., 2021) model.

Original: 3. I NFERENCE
Revised: 3. I NFERENCE

Original: At inference time we go over each time series i ∈ D train and feed the last context sized window (except for the last entry) to the encoder and the very last entry to decoder to obtain the parameters of the distribution of the next time point.
Revised: At inference time we go over each time series i ∈ D train and feed the last context sized window (except for the last entry) to the encoder and the very last entry to decoder to obtain the parameters of the distribution of the next time point.

Original: We can now sample one or more values from this distribution and feed it back to the decoder to obtain samples for each time point of our desired horizon of P time steps.
Revised: We can now sample one or more values from this distribution and feed it back to the decoder to obtain samples for each time point of our desired horizon of P time steps.

Original: Note that we only need to run the encoder once in order to predict and can repeat tensors in the batch dimension to obtain many samples from the distribution in parallel.
Revised: Note that we only need to run the encoder once in order to predict and can repeat tensors in the batch dimension to obtain many samples from the distribution in parallel.

Original: If a point forecast is required then we can evaluate the empirical mean or median at each time point of the prediction.
Revised: If a point forecast is required then we can evaluate the empirical mean or median at each time point of the prediction.

Original: Unlike some generative modeling models, here we do not sample from a reduced temperature distribution to obtain high quality samples as we are interested in the empirical data distribution of the next time point conditioned on the past as well as covariates.
Revised: Unlike some generative modeling models, here we do not sample from a reduced temperature distribution to obtain high quality samples as we are interested in the empirical data distribution of the next time point conditioned on the past as well as covariates.

Original: 3. C OVARIATES
Revised: 3. C OVARIATES

Original: Positional encoding give the Transformer the ability to encode positional information of sequences when needed since Attention is a permutation equivariant layer.
Revised: Positional encoding give the Transformer the ability to encode positional information of sequences when needed since Attention is a permutation equivariant layer.

Original: In the time series setting we can naturally create positional encodings like Rotary Positional Embedding (RoPE)
Revised: In the time series setting we can naturally create positional encodings like Rotary Positional Embedding (RoPE)

Original: (Su et al., 2021) via date-time covariates.
Revised: (Su et al., 2021) via date-time covariates.

Original: More specifically, for a particular time point t , depending on the frequency of the time series i , we can create hour-of-day, day-of-week, week-of-month, etc.
Revised: More specifically, for a particular time point t , depending on the frequency of the time series i , we can create hour-of-day, day-of-week, week-of-month, etc.

Original: features as a vector we denote by c it .
Revised: features as a vector we denote by c it .

Original: Due to their temporal nature we can build these covariates for all future time points we wish to forecast for.
Revised: Due to their temporal nature we can build these covariates for all future time points we wish to forecast for.

Original: Additional covariates can be constructed by considering the running means, the age of a time series as well as embedding the identity i of each time series in a dataset via Embedding layers, as done in the DeepAR method.
Revised: Additional covariates can be constructed by considering the running means, the age of a time series as well as embedding the identity i of each time series in a dataset via Embedding layers, as done in the DeepAR method.

Original: 3. S CALING
Revised: 3. S CALING

Original: As detailed in the Salinas et al.
Revised: As detailed in the Salinas et al.

Original: (2019b), time series data can be of an arbitrary numerical magnitude within a dataset.
Revised: (2019b), time series data can be of an arbitrary numerical magnitude within a dataset.

Original: This is unlike the vision, NLP or even audio modalities, and so in order to train a shared model over potentially very different time series we calculate the mean value of the signal within its context window and divide the signal with it to normalize it.
Revised: This is unlike the vision, NLP or even audio modalities, and so in order to train a shared model over potentially very different time series we calculate the mean value of the signal within its context window and divide the signal with it to normalize it.

Original: This context window scale value is kept as a covariate and more importantly the model’s output distribution is transformed back to the original scale via it during training and inference to calculate the log-probabilities or to sample from respectively.
Revised: This context window scale value is kept as a covariate and more importantly the model’s output distribution is transformed back to the original scale via it during training and inference to calculate the log-probabilities or to sample from respectively.

Original: If the scaling cannot be done in the output distribution’s parameter space one can also do it in the data space after sampling.
Revised: If the scaling cannot be done in the output distribution’s parameter space one can also do it in the data space after sampling.

Original: All deep learning based methods in Section 4 incorporate this heuristic.
Revised: All deep learning based methods in Section 4 incorporate this heuristic.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test the performance of VQ-TR for the forecasting task in this section with respect to a number of methods on a number of open datasets.
Revised: We test the performance of VQ-TR for the forecasting task in this section with respect to a number of methods on a number of open datasets.

Original: [Table]
Revised: [Table]

Original: For our experiments we use the following open datasets: Exchange (Lai et al., 2018),
Revised: For our experiments we use the following open datasets: Exchange (Lai et al., 2018),

Original: Solar (Lai et al., 2018), Elec. 1 , Traffic 2 , Taxi 3 , and Wikipedia 4 preprocessed exactly as in Salinas et al. (2019a), with their properties listed in Table 1.
Revised: Solar (Lai et al., 2018), Elec. 1 , Traffic 2 , Taxi 3 , and Wikipedia 4 preprocessed exactly as in Salinas et al. (2019a), with their properties listed in Table 1.

Original: As can be noted in the table, we do not need to normalize scales for the Traffic dataset.
Revised: As can be noted in the table, we do not need to normalize scales for the Traffic dataset.

Original: From the names of the datasets, we see that we cover a number of time series domains including finance, weather, energy, logistics and page-views
Revised: From the names of the datasets, we see that we cover a number of time series domains including finance, weather, energy, logistics and page-views

Original: We will compare VQ-TR with the following deep learning baseline probabilistic univariate models
Revised: We will compare VQ-TR with the following deep learning baseline probabilistic univariate models

Original: /datasets
Revised: /datasets

Original: [Table]
Revised: [Table]

Original: IQN-RNN 0.
Revised: IQN-RNN 0.

Original: MQCNN 0.
Revised: MQCNN 0.

Original: VQ-AR-iqn 0.
Revised: VQ-AR-iqn 0.

Original: VQ-TR-iqn 0.
Revised: VQ-TR-iqn 0.

Original: Elec.
Revised: Elec.

Original: SQF-RNN0.
Revised: SQF-RNN0.

Original: DeepAR-t 0.
Revised: DeepAR-t 0.

Original: ETS 0.
Revised: ETS 0.

Original: IQN-RNN 0.
Revised: IQN-RNN 0.

Original: MQCNN 0.
Revised: MQCNN 0.

Original: VQ-AR-t 0.
Revised: VQ-AR-t 0.

Original: VQ-TR
Revised: VQ-TR

Original: [Table]
Revised: [Table]

Original: t 0.
Revised: t 0.

Original: Traffic
Revised: Traffic

Original: SQF-RNN0.
Revised: SQF-RNN0.

Original: DeepAR-t 0.
Revised: DeepAR-t 0.

Original: ETS 0.
Revised: ETS 0.

Original: [Table]
Revised: [Table]

Original: MQCNN 1.
Revised: MQCNN 1.

Original: VQ-AR-t 0.
Revised: VQ-AR-t 0.

Original: VQ-TR-t 0.
Revised: VQ-TR-t 0.

Original: Taxi
Revised: Taxi

Original: SQF
Revised: SQF

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ETS 1.
Revised: ETS 1.

Original: IQN
Revised: IQN

Original: [Table]
Revised: [Table]

Original: MQCNN 1.
Revised: MQCNN 1.

Original: VQ
Revised: VQ

Original: [Table]
Revised: [Table]

Original: AR
Revised: AR

Original: [Table]
Revised: [Table]

Original: VQ
Revised: VQ

Original: [Table]
Revised: [Table]

Original: TR-t 0.
Revised: TR-t 0.

Original: Wiki.
Revised: Wiki.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: • DeepAR Salinas et al.
Revised: • DeepAR Salinas et al.

Original: (2019b): an RNN based probabilistic model which learns the parameters of some chosen distribution for the next time point; • MQCNN Wen et al.
Revised: (2019b): an RNN based probabilistic model which learns the parameters of some chosen distribution for the next time point; • MQCNN Wen et al.

Original: (2017): a Convolutional Neural Network model which outputs chosen quantiles of the forecast upon which we regress the ground truth via Quantile loss;
Revised: (2017): a Convolutional Neural Network model which outputs chosen quantiles of the forecast upon which we regress the ground truth via Quantile loss;

Original: • SQF-RNN Gasthaus et al.
Revised: • SQF-RNN Gasthaus et al.

Original: (2019): an RNN based non-parametric method which models the quantiles via linear splines and also regresses the Quantile loss; • IQN-RNN Gouttes et al.
Revised: (2019): an RNN based non-parametric method which models the quantiles via linear splines and also regresses the Quantile loss; • IQN-RNN Gouttes et al.

Original: (2021): combines an RNN model with an Implicit Quantile Network (IQN) Dabney et al.
Revised: (2021): combines an RNN model with an Implicit Quantile Network (IQN) Dabney et al.

Original: (2018) head to learn the distribution similar to SQF-RNN ; • VQ-AR Rasul et al.
Revised: (2018) head to learn the distribution similar to SQF-RNN ; • VQ-AR Rasul et al.

Original: (2022): an RNN based encoder-decoder model which quantizes its input via a VQ;
Revised: (2022): an RNN based encoder-decoder model which quantizes its input via a VQ;

Original: as well as the classical ETS Hyndman & Khandakar (2008) which is an exponential smoothing method using weighted averages of past observations with exponentially decaying weights as the observations get older together with Gaussian additive errors (E) modeling trend (T) and seasonality (S) effects separately.
Revised: as well as the classical ETS Hyndman & Khandakar (2008) which is an exponential smoothing method using weighted averages of past observations with exponentially decaying weights as the observations get older together with Gaussian additive errors (E) modeling trend (T) and seasonality (S) effects separately.

Original: We follow the recommendations of the M4 competition Makridakis et al.
Revised: We follow the recommendations of the M4 competition Makridakis et al.

Original: (2020) for reporting forecasting performance metrics.
Revised: (2020) for reporting forecasting performance metrics.

Original: In this regard, we report the mean scale interval score Gneiting &
Revised: In this regard, we report the mean scale interval score Gneiting &

Original: Raftery (2007) (MSIS 5 ) for a 95% prediction interval, the 50-th and 90-th quantile percentile loss (QL50 and QL90 respectively), as well as the Continuous Ranked Probability Score (CRPS) Gneiting & Raftery (2007); Matheson & Winkler (1976) score.
Revised: Raftery (2007) (MSIS 5 ) for a 95% prediction interval, the 50-th and 90-th quantile percentile loss (QL50 and QL90 respectively), as well as the Continuous Ranked Probability Score (CRPS) Gneiting & Raftery (2007); Matheson & Winkler (1976) score.

Original: The CRPS is a proper scoring rule which measures the compatibility of a predicted cumulative distribution function (CDF) F with the groundtruth samples x as
Revised: The CRPS is a proper scoring rule which measures the compatibility of a predicted cumulative distribution function (CDF) F with the groundtruth samples x as

Original: [Equation]
Revised: [Equation]

Original: where I { x ≤ y } is 1 if x ≤ y and 0 otherwise.
Revised: where I { x ≤ y } is 1 if x ≤ y and 0 otherwise.

Original: We approximate the CDF via empirical samples at each time point and the final metric is averaged over the prediction horizon and time series of a dataset.
Revised: We approximate the CDF via empirical samples at each time point and the final metric is averaged over the prediction horizon and time series of a dataset.

Original: The point-forecasting performance of models is measured by the normalized root mean square error (NRMSE), the mean absolute scaled error (MASE) Hyndman & Koehler (2006), and the symmetric mean absolute percentage error (sMAPE) Makridakis (1993).
Revised: The point-forecasting performance of models is measured by the normalized root mean square error (NRMSE), the mean absolute scaled error (MASE) Hyndman & Koehler (2006), and the symmetric mean absolute percentage error (sMAPE) Makridakis (1993).

Original: For pointwise metrics, we use sampled medians with the exception of NRMSE, where we take the mean over our prediction samples.
Revised: For pointwise metrics, we use sampled medians with the exception of NRMSE, where we take the mean over our prediction samples.

Original: The results of our extensive experiments are detailed in Table 2.
Revised: The results of our extensive experiments are detailed in Table 2.

Original: As can be seen VQ-TR performs competitively with respect to the methods compared, where the models have been trained using the hyperparameters from their respective papers using Student-T ( -t ), Negative Binomial ( -nb ) or
Revised: As can be seen VQ-TR performs competitively with respect to the methods compared, where the models have been trained using the hyperparameters from their respective papers using Student-T ( -t ), Negative Binomial ( -nb ) or

Original: Implicit Quantile Network ( -iqn ) emission heads.
Revised: Implicit Quantile Network ( -iqn ) emission heads.

Original: In particular for VQ-TR we can afford to use a larger context length of C = 20 × P , where P is the prediction horizon for each dataset, with the total number of encoder layers N = 2 and decoder layers M = 6 .
Revised: In particular for VQ-TR we can afford to use a larger context length of C = 20 × P , where P is the prediction horizon for each dataset, with the total number of encoder layers N = 2 and decoder layers M = 6 .

Original: For training we use J =codebook vectors, batch size of 256 for 20 epochs using the Adam (Kingma & Ba, 2015) optimizer with default parameters, and a learning rate of 0 .
Revised: For training we use J =codebook vectors, batch size of 256 for 20 epochs using the Adam (Kingma & Ba, 2015) optimizer with default parameters, and a learning rate of 0 .

Original: At inference time we sample S = 100 times for each time point and feed these samples in parallel via the batch dimension autoregressively through the decoder to produce the reported metrics.
Revised: At inference time we sample S = 100 times for each time point and feed these samples in parallel via the batch dimension autoregressively through the decoder to produce the reported metrics.

Original: The full source code will be made available after the review period.
Revised: The full source code will be made available after the review period.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: This method separates the length of the input sequence from the computation of the attention block by using a discrete set of latents.
Revised: This method separates the length of the input sequence from the computation of the attention block by using a discrete set of latents.

Original: This strategy of reducing the computational cost is similar to the
Revised: This strategy of reducing the computational cost is similar to the

Original: Perceiver (Hawthorne et al., 2022), Set Transformer (Lee et al., 2019), Luna (Ma et al., 2021) and Compression Transformer (Rae et al., 2020) models.
Revised: Perceiver (Hawthorne et al., 2022), Set Transformer (Lee et al., 2019), Luna (Ma et al., 2021) and Compression Transformer (Rae et al., 2020) models.

Original: Perceiver-AR is the closest related method, however it is a decoder only architecture and thus at inference time with many parallel samples for the probabilistic forecasting usecase, we have to run the cross-attention over a large context window for P times causing both memory and computation bottlenecks.
Revised: Perceiver-AR is the closest related method, however it is a decoder only architecture and thus at inference time with many parallel samples for the probabilistic forecasting usecase, we have to run the cross-attention over a large context window for P times causing both memory and computation bottlenecks.

Original: The use of VQ in sequential generative modeling has been explored in Audio/Speech setting Dhariwal et al.
Revised: The use of VQ in sequential generative modeling has been explored in Audio/Speech setting Dhariwal et al.

Original: (2020); Zeghidour et al.
Revised: (2020); Zeghidour et al.

Original: ; Baevski et al.
Revised: ; Baevski et al.

Original: (2020) where typically a VQ-VAE is trained on the data and then a generative model trained on these learned latents separately.
Revised: (2020) where typically a VQ-VAE is trained on the data and then a generative model trained on these learned latents separately.

Original: The use of VQ in time series forecasting problem has been explored in the VQ-AR (Rasul et al., 2022) model
Revised: The use of VQ in time series forecasting problem has been explored in the VQ-AR (Rasul et al., 2022) model

Original: however it uses an RNN to encode the history of a time series to a discrete latent which is then used for the forecasting decoder RNN.
Revised: however it uses an RNN to encode the history of a time series to a discrete latent which is then used for the forecasting decoder RNN.

Original: In contrast this work incorporates the VQ module as part of an approximate Attention block to mitigate issues of large temporal contexts over which to forecast over.
Revised: In contrast this work incorporates the VQ module as part of an approximate Attention block to mitigate issues of large temporal contexts over which to forecast over.

Original:  S UMMARY AND D ISCUSSION
Revised:  S UMMARY AND D ISCUSSION

Original: We have presented VQ-TR a novel architecture which scales linearly with the encoder sequence size as a probabilistic forecasting model and demonstrated its performance against competitive models on a number of open datasets.
Revised: We have presented VQ-TR a novel architecture which scales linearly with the encoder sequence size as a probabilistic forecasting model and demonstrated its performance against competitive models on a number of open datasets.

Original: VQ-TR reports good performance at test time and we can control the trade-off of computation and memory use by increasing or decreasing the number of discrete latents
Revised: VQ-TR reports good performance at test time and we can control the trade-off of computation and memory use by increasing or decreasing the number of discrete latents

Original: J .
Revised: J .

Original: As the reader might guess, this architecture can also work in other sequential modeling usecases and in future work we would like investigate the performance of VQ-TR for NLP, Audio or Vision based problems.
Revised: As the reader might guess, this architecture can also work in other sequential modeling usecases and in future work we would like investigate the performance of VQ-TR for NLP, Audio or Vision based problems.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We acknowledge and thank the authors and contributors of the many open source libraries that were used in this work, in particular: GluonTS (Alexandrov et al., 2020), Vector Quantize PyTorch (Wang & Contributors, 2022), NumPy (Harris et al., 2020), Pandas (Pandas development team, 2020), Matplotlib (Hunter, 2007), Jupyter (Kluyver et al., 2016) and PyTorch (Paszke et al., 2019).
Revised: We acknowledge and thank the authors and contributors of the many open source libraries that were used in this work, in particular: GluonTS (Alexandrov et al., 2020), Vector Quantize PyTorch (Wang & Contributors, 2022), NumPy (Harris et al., 2020), Pandas (Pandas development team, 2020), Matplotlib (Hunter, 2007), Jupyter (Kluyver et al., 2016) and PyTorch (Paszke et al., 2019).

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Time series models considered in this work have been trained on open datasets which do not contain any personal information or ways of deducing personal information from them.
Revised: Time series models considered in this work have been trained on open datasets which do not contain any personal information or ways of deducing personal information from them.

Original: All the models considered however can potentially be used, when trained with personal information, to predict behaviour or deduce private information like location for example, which could be potentially risky.
Revised: All the models considered however can potentially be used, when trained with personal information, to predict behaviour or deduce private information like location for example, which could be potentially risky.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We will open source the code to the paper after the review period.
Revised: We will open source the code to the paper after the review period.

Original: The dataset and preprocessing used in this work is based on open datasets with predefined preprocessing provided by GluonTS (Alexandrov et al., 2020).
Revised: The dataset and preprocessing used in this work is based on open datasets with predefined preprocessing provided by GluonTS (Alexandrov et al., 2020).

Original: All the experiments have their hyperparameters in the appropriate Jupyter (Kluyver et al., 2016) notebooks.
Revised: All the experiments have their hyperparameters in the appropriate Jupyter (Kluyver et al., 2016) notebooks.

Original: A A PPENDIX A. E XPERIMENTS ON T RANSFORMER BASED M ODELS
Revised: E XPERIMENTS ON T RANSFORMER BASED M ODELS

Original: We further compare against a range of Transformer based models and include the results in  the following:
Revised: We further compare against a range of Transformer based models and include the results in Table 3. We compare, apart from the vanilla Transformer , against:

Revised: • TFT Lim et al.

Revised: (2021): an auto-regressive attention based Seq-to-Seq model with variable selection network for selecting relevant inputs; • Informer Zhou et al.

Revised: (2021): an efficient transformer and full horizon predictor model; • Autoformer Wu et al.

Revised: (2021): a transformer which decomposition the trend and seasonal components during the forecasting process together with series-wise auto-Correlation mechanism;

Original: [Table]
Revised: Method M e m o r y M i B ) V Q - T R

Revised: T r a n s f o r m e r

Revised: A u t o f o r m e r

Revised: E T S F o r m e r

Revised: H o p f i e l d

Revised: I n f o r m e r

Revised: R e f o r m e r

Revised: TFT

Revised: Lon g f o r m e r

Revised: P e

Revised: r

Revised: f

Revised: o r m e r

Original: Informer-t 0.
Revised: • ETSformer Woo et al. (2022): a transformer architecture which adds the principle of exponential smoothing and frequency attention in the attention mechanism; • Hopfield Ramsauer et al. (2021): a modern Hopfield network with continuous state which generalizes attention; • Longformer Beltagy et al. (2020): a local attention model with sliding window attention; • Performer Choromanski et al.

Original: Autoformer-t 0.
Revised: (2021): a transformer model which estimates regular fullrank attention by using linear space/compute complexity; [Figure] [Figure] To analyse the amount of memory for the different Transformer based methods, we plot this metric in Figure 2, when we train on Traffic using the same number of heads (2), encoding/decoding layers (2), feed-forward dimensions (16), context length (20 times prediction length) and batch size.

Original: ETSformer-t 0.
Revised: [Table] Solar Trans-t 0. 1.499 Tft-t 0.

Revised: 1.950 Informer-t 0.

Revised: 1.254 Autoformer-t 0.

Original:  Hopfield-t 0. 
Revised: 2.317 ETSformer-t 0. 1.166 Hopfield-t 0. 1.505 Longformer-t 0. 1.314 Performer-t 0.

Original:  VQ-TR-iqn 0.
Revised: 1.474 VQ-TR-iqn 0.

Original: Elec.
Revised: Elec.

Original: Trans-t 0.
Revised: Trans-t 0.

Original:  Tft-t 0.
Revised: 0.853 Tft-t 0.

Original:  Informer-t 0.
Revised: 0.676 Informer-t 0.

Original:  Autoformer-t 0.
Revised: 0.788 Autoformer-t 0.

Original:  ETSformer-t 0.
Revised: 1.00 ETSformer-t 0.

Original:  Hopfield-t 0.
Revised: 0.904 Hopfield-t 0.

Original:  Reformer-t 0. 
Revised: 0.736 Reformer-t 0. 0.835 Linformer-t 0. 1.024 Longformer-t 0. 3.465 Performer-t 0.

Original:  VQ-TR-t 0.
Revised: 2.470 VQ-TR-t 0.

Original: Traffic
Revised: Traffic

Original: Trans-t 0.
Revised: Trans-t 0.

Original:  TFT-t 0.
Revised: 1.300 TFT-t 0.

Original:  Informer-t 0.
Revised: 0.648 Informer-t 0.

Original: 0.528t Autoformer-t 0.
Revised: 0.528 Autoformer-t 0.

Original:  ETSformer-t 0.
Revised: 0.901 ETSformer-t 0.

Original:  Hopfield-t 0. 
Revised: 0.783 Hopfield-t 0. 0.534 Longformer-t 0. 1.651 Performer-t 0.

Original:  VQ-TR-t 0.
Revised: 1.736 VQ-TR-t 0.

Original: Taxi
Revised: Taxi

Original: Trans-nb 0.
Revised: Trans-nb 0.

Original:  Tft-nb 0.
Revised: 0.790 Tft-nb 0.

Original: Informer-t 0.
Revised: 0.767 Informer-nb 0.

Original:  Autoformer-nb 0.  ETSformer-nb 0.
Revised: 0.825 Autoformer-nb 0. 0.916 ETSformer-nb 0.

Original:  Hopfield-nb 0. 
Revised: 0.797 Hopfield-nb 0. 0.850 Linformer-t 0. 1.855 Longformer-nb 0. 0.937 Performer-nb 0.

Original:  VQ-TR-t 0.
Revised: 0.954 VQ-TR-t 0.

Original: Wiki.
Revised: Wiki.

Original: Trans-nb 0.
Revised: Trans-nb 0.

Original:  Tft-nb 0.
Revised: 1.837 Tft-nb 0.

Original: Informer-t 0.
Revised: 1.566 Informer-nb 0.

Original: Hopfield-nb 2. VQ-TR-iqn 0.
Revised: [Table]