Original: S PECTRA N ET : MULTIVARIATE FORECASTING AND IMPUTATION UNDER DISTRIBUTION SHIFTS AND MISSING DATA
Revised: S PECTRA N ET : MULTIVARIATE FORECASTING AND IMPUTATION UNDER DISTRIBUTION SHIFTS AND MISSING DATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work, we tackle two widespread challenges in real applications for timeseries forecasting that have been largely understudied: distribution shifts and missing data.
Revised: In this work, we tackle two widespread challenges in real applications for timeseries forecasting that have been largely understudied: distribution shifts and missing data.

Original: We propose SpectraNet , a novel multivariate time-series forecasting model that dynamically infers a latent space spectral decomposition to capture current temporal dynamics and correlations on the recent observed history.
Revised: We propose SpectraNet , a novel multivariate time-series forecasting model that dynamically infers a latent space spectral decomposition to capture current temporal dynamics and correlations on the recent observed history.

Original: A Convolution Neural Network maps the learned representation by sequentially mixing its components and refining the output.
Revised: A Convolution Neural Network maps the learned representation by sequentially mixing its components and refining the output.

Original: Our proposed approach can simultaneously produce forecasts and interpolate past observations and can, therefore, greatly simplify production systems by unifying imputation and forecasting tasks into a single model.
Revised: Our proposed approach can simultaneously produce forecasts and interpolate past observations and can, therefore, greatly simplify production systems by unifying imputation and forecasting tasks into a single model.

Original: SpectraNet achieves SoTA performance simultaneously on both tasks on five benchmark datasets, compared to forecasting and imputation models, with up to 92% fewer parameters and comparable training times.
Revised: SpectraNet achieves SoTA performance simultaneously on both tasks on five benchmark datasets, compared to forecasting and imputation models, with up to 92% fewer parameters and comparable training times.

Original: On settings with up to 80% missing data, SpectraNet has average performance improvements of almost 50% over the second-best alternative.
Revised: On settings with up to 80% missing data, SpectraNet has average performance improvements of almost 50% over the second-best alternative.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multivariate time-series forecasting is an essential task in a wide range of domains.
Revised: Multivariate time-series forecasting is an essential task in a wide range of domains.

Original: Forecasts are a key input to optimize the production and distribution of goods (B ¨ ose et al., 2017), predict healthcare patient outcomes (Chen et al., 2015), plan electricity production (Olivares et al., 2022), build financial portfolios (Emerson et al., 2019), among other examples.
Revised: Forecasts are a key input to optimize the production and distribution of goods (B ¨ ose et al., 2017), predict healthcare patient outcomes (Chen et al., 2015), plan electricity production (Olivares et al., 2022), build financial portfolios (Emerson et al., 2019), among other examples.

Original: Due to its high potential benefits, researchers have dedicated many efforts to improving the capabilities of forecasting models, with breakthroughs in model architectures and performance (Benidis et al., 2022).
Revised: Due to its high potential benefits, researchers have dedicated many efforts to improving the capabilities of forecasting models, with breakthroughs in model architectures and performance (Benidis et al., 2022).

Original: The main focus of research in multivariate forecasting has been on accuracy and scalability, to which the present paper contributes.
Revised: The main focus of research in multivariate forecasting has been on accuracy and scalability, to which the present paper contributes.

Original: In addition, we identify two widespread challenges for real applications which have been largely understudied: distribution shifts and missing data .
Revised: In addition, we identify two widespread challenges for real applications which have been largely understudied: distribution shifts and missing data .

Original: We refer to distribution shifts as changes in the time-series behavior.
Revised: We refer to distribution shifts as changes in the time-series behavior.

Original: In particular, we focus on discrepancies in distribution between the train and test data, which can considerably degrade the accuracy (Kuznetsov & Mohri, 2014; Du et al., 2021; Xu et al., 2022; Ivanovic et al., 2022).
Revised: In particular, we focus on discrepancies in distribution between the train and test data, which can considerably degrade the accuracy (Kuznetsov & Mohri, 2014; Du et al., 2021; Xu et al., 2022; Ivanovic et al., 2022).

Original: This has become an increasing problem in recent years with the COVID-19 pandemic, which disrupted all aspects of human activities.
Revised: This has become an increasing problem in recent years with the COVID-19 pandemic, which disrupted all aspects of human activities.

Original: Missing values is a generalized problem.
Revised: Missing values is a generalized problem.

Original: Some common causes include faulty sensors, the impossibility of gathering data, and misplacement of information.
Revised: Some common causes include faulty sensors, the impossibility of gathering data, and misplacement of information.

Original: As we demonstrate in our experiments, these challenges hinder the performance of current state-of-the-art (SoTA), limiting their use in applications where these problems are predominant.
Revised: As we demonstrate in our experiments, these challenges hinder the performance of current state-of-the-art (SoTA), limiting their use in applications where these problems are predominant.

Original: In this work, we propose SpectraNet , a novel multivariate forecasting model that achieves SoTA performance in benchmark datasets and is also intrinsically robust to distribution shifts and extreme cases of missing data.
Revised: In this work, we propose SpectraNet , a novel multivariate forecasting model that achieves SoTA performance in benchmark datasets and is also intrinsically robust to distribution shifts and extreme cases of missing data.

Original: SpectraNet achieves its high accuracy and robustness by dynamically inferring a latent vector projected on a temporal basis, a process we name latent space spectral decomposition (LSSD).
Revised: SpectraNet achieves its high accuracy and robustness by dynamically inferring a latent vector projected on a temporal basis, a process we name latent space spectral decomposition (LSSD).

Original: A series of convolution layers then synthesize both the reference window, which is used to infer the latent vectors and the forecast window.
Revised: A series of convolution layers then synthesize both the reference window, which is used to infer the latent vectors and the forecast window.

Original: To the best of our knowledge, SpectraNet is also the first solution that can simultaneously forecast the future values of a multivariate time series and accurately impute the past missing data.
Revised: To the best of our knowledge, SpectraNet is also the first solution that can simultaneously forecast the future values of a multivariate time series and accurately impute the past missing data.

Original: In 𝑧 ! 𝑧 " . 𝑧 # × ⋮ 𝑬 = 𝒛 ∗ 𝑩 𝑠 # /2 𝑑 𝑩 𝒛 ∗  𝑳𝑺𝑺𝑫 𝑪𝑵𝑵 𝒉𝑠 # 𝒉𝑠 # / 𝑛 & 𝑛 ’ 𝑠 # 𝑀
Revised: × ⋮ ( = * ∗ , - . /2, * ∗  7- .

Original: ConvTranspose1d
Revised: Transposed Convolution

Original: Referencewindow Forecastingwindow 𝑡 +𝐻 #𝒀 𝒕"𝑳:𝒕%𝑯 𝑡 −𝐿 𝑡 𝑌 !
Revised: Referencewindow Forecastingwindow < +> ?@ AB2:ADE < −G <

Original: 𝑌 "  ⋮
Revised: H " H I ⋮

Original: BN + ReLU
Revised: BN + ReLU

Original: The Convolution Network (CNN) generates the time-series window by sequentially mixing the components of the embedding and refining the output.
Revised: The Convolution Network (CNN) generates the time-series window by sequentially mixing the components of the embedding and refining the output.

Original:  practice, imputation models are first used to fill the missing information for all downstream tasks, including forecasting.
Revised: In practice, imputation models are first used to fill the missing information for all downstream tasks, including forecasting.

Original: SpectraNet can greatly simplify production systems by unifying imputation and forecasting tasks into a single model.
Revised: SpectraNet can greatly simplify production systems by unifying imputation and forecasting tasks into a single model.

Original: The main contributions are:
Revised: The main contributions are:

Original: • Latent Vector Inference : methodology to dynamically capture current dynamics of the target time-series into a latent space, replacing parametric encoders.
Revised: • Latent Vector Inference : methodology to dynamically capture current dynamics of the target time-series into a latent space, replacing parametric encoders.

Original: • Latent Space Spectral Decomposition: representation of a multivariate time-series window on a shared latent space with temporal dynamics.
Revised: • Latent Space Spectral Decomposition: representation of a multivariate time-series window on a shared latent space with temporal dynamics.

Original: • SpectraNet : novel multivariate forecasting model that simultaneously imputes missing data and forecasts future values, with SoTA performance on several benchmark datasets and demonstrated robustness to distribution shifts and missing values.
Revised: • SpectraNet : novel multivariate forecasting model that simultaneously imputes missing data and forecasts future values, with SoTA performance on several benchmark datasets and demonstrated robustness to distribution shifts and missing values.

Original: We will make our code publicly available upon acceptance.
Revised: We will make our code publicly available upon acceptance.

Original: The remainder of this paper is structured as follows.
Revised: The remainder of this paper is structured as follows.

Original: Section 2 introduces notation and the problem definition, Section 3 presents our method, Section 4 describes and presents our empirical findings.
Revised: Section 2 introduces notation and the problem definition, Section 3 presents our method, Section 4 describes and presents our empirical findings.

Original: Finally, Section 5 concludes the paper.
Revised: Finally, Section 5 concludes the paper.

Original: The literature review is included in A.1.
Revised: The literature review is included in A.1.

Original:  N OTATION AND P ROBLEM DEFINITION
Revised:  N OTATION AND P ROBLEM DEFINITION

Original: We introduce a new notation that we believe is lighter than the standard notation while being intuitive and formally correct.
Revised: We introduce a new notation that we believe is lighter than the standard notation while being intuitive and formally correct.

Original: Let Y ∈ R M × T be a multivariate time-series with M features and T timestamps.
Revised: Let Y ∈ R M × T be a multivariate time-series with M features and T timestamps.

Original: Let Y a : b ∈ R M × ( b − a ) be the observed values for the interval [ a, b ) , that is, Y 0:
Revised: Let Y a : b ∈ R M × ( b − a ) be the observed values for the interval [ a, b ) , that is, Y 0:

Original: t is the set of t observations of Y from timestamp 0 to timestamp t − 1 while Y t : t
Revised: t is the set of t observations of Y from timestamp 0 to timestamp t − 1 while Y t : t

Original: + H is the set of H observations of Y from timestamp t to timestamp t
Revised: + H is the set of H observations of Y from timestamp t to timestamp t

Original: + H − 1 .
Revised: + H − 1 .

Original: Let y m,t ∈ R be the value of feature m at timestamp t .
Revised: Let y m,t ∈ R be the value of feature m at timestamp t .

Original: In this work we consider the multivariate point forecasting task, which consists of predicting the future values of a multivariate time-series sequence based on past observations.
Revised: In this work we consider the multivariate point forecasting task, which consists of predicting the future values of a multivariate time-series sequence based on past observations.

Original: The main task of a model F Θ with parameters Θ at a timestamp t , is to produce forecasts for the future H values, denoted by ˆY t : t + H , based on the previous history Y 0: t .
Revised: The main task of a model F Θ with parameters Θ at a timestamp t , is to produce forecasts for the future H values, denoted by ˆY t : t + H , based on the previous history Y 0: t .

Original: [Equation]
Revised: [Equation]

Original: For the imputation task, to impute a missing value y m,t , models are not constrained to only use past observations.
Revised: For the imputation task, to impute a missing value y m,t , models are not constrained to only use past observations.

Original: Moreover, they are evaluated on how well they approximate only the missing values.
Revised: Moreover, they are evaluated on how well they approximate only the missing values.

Original: We evaluate the performance with two common metrics used in the literature, mean squared error (MSE) and mean absolute error (MAE), given by equation 2 (Hyndman & Athanasopoulos, 2018).
Revised: We evaluate the performance with two common metrics used in the literature, mean squared error (MSE) and mean absolute error (MAE), given by equation 2 (Hyndman & Athanasopoulos, 2018).

Original: [Equation]
Revised: [Equation]

Original:  S PECTRA N ET
Revised:  S PECTRA N ET

Original: We start the description of our approach with a general outline of the model and explain each major component in detail in the following subsections.
Revised: We start the description of our approach with a general outline of the model and explain each major component in detail in the following subsections.

Original: The overall architecture is illustrated in Figure 1.
Revised: The overall architecture is illustrated in Figure 1.

Original: SpectraNet is a top-down model that generates a multivariate time-series window of fixed size s w =
Revised: SpectraNet is a top-down model that generates a multivariate time-series window of fixed size s w =

Original: L
Revised: L

Original: + H , where L is the length of the reference window and H is the forecast horizon, from a latent vector z ∈ R d .
Revised: + H , where L is the length of the reference window and H is the forecast horizon, from a latent vector z ∈ R d .

Original: To produce the forecasts at timestamp t , the model first infers the optimal latent vector on the reference window , consisting of the last L values, Y t − L : t , by minimizing the reconstruction error .
Revised: To produce the forecasts at timestamp t , the model first infers the optimal latent vector on the reference window , consisting of the last L values, Y t − L : t , by minimizing the reconstruction error .

Original: This inference step is the main difference between our approach to existing models, which map the input into an embedding or latent space using an encoder network.
Revised: This inference step is the main difference between our approach to existing models, which map the input into an embedding or latent space using an encoder network.

Original: The model generates the full time-series window s w , which includes the forecast window Y t : t + H , with a spectral decomposition and a Convolutional Neural Network (CNN).
Revised: The model generates the full time-series window s w , which includes the forecast window Y t : t + H , with a spectral decomposition and a Convolutional Neural Network (CNN).

Original: The main steps of SpectraNet are given by
Revised: The main steps of SpectraNet are given by

Original: [Equation]
Revised: [Equation]

Original: where z ∗ is the inferred latent vector, L is a reconstruction error metric, and ˆ Y t − L : t ( z ) is given by,
Revised: where z ∗ is the inferred latent vector, L is a reconstruction error metric, and ˆ Y t − L : t ( z ) is given by,

Original: [Equation]
Revised: [Equation]

Original: where LSSD ( latent space spectral decomposition ) is a basis expansion operation of z over the predefined temporal basis B to produce a temporal embedding E ∈ R d × d t , and CNN is a Top-Down
Revised: where LSSD ( latent space spectral decomposition ) is a basis expansion operation of z over the predefined temporal basis B to produce a temporal embedding E ∈ R d × d t , and CNN is a Top-Down

Original: Convolutional Neural Network with learnable parameters Θ .
Revised: Convolutional Neural Network with learnable parameters Θ .

Original: The CNN simultaneously produces both the reconstruction of the past reference window ˆ Y t − L : t , used to find the optimal latent vector for the full window and the forecast ˆ Y t : t + H .
Revised: The CNN simultaneously produces both the reconstruction of the past reference window ˆ Y t − L : t , used to find the optimal latent vector for the full window and the forecast ˆ Y t : t + H .

Original: 3. L ATENT VECTORS INFERENCE
Revised: 3. L ATENT VECTOR INFERENCE

Original: The main difference between our approach and existing models is  the inference of the  latent vectors instead of relying on encoders.
Revised: The proposed latent vector inference is based on the Alternating back-propagation algorithm (ABP) (Han et al., 2017) for training generative models without encoders. A single generator (decoder) architecture is trained by maximizing the observed likelihood directly. To achieve this, at every step, ABP samples latent vectors from the posterior distribution P ( Z | Y ) using MCMC methods such as Langevin dynamics.

Revised: Generative models trained with ABP demonstrated superior performance in recovering missing segments of images and videos over Variational Autoencoders (VAE) and

Original: The latent vectors are inferred by minimizing the MSE between the observed values Y t − L : t and the  reconstruction ˆ Y t − L : t ( z ) on the reference window.
Revised: Generative Adversarial Networks (GAN). To our knowledge, SpectraNet is the first approach that uses and extends the latent vector inference principle of ABP for time-series forecasting. We reformulate the posterior distribution sampling of the latent vectors as a non-convex minimization problem presented in 3, which aims to minimize the generator’s reconstruction error on the reference window Y t − L : t . We use the mean square error (MSE) as the reconstruction loss since it is differentiable, fast to compute and theoretically founded.

Original: By doing this, the model dynamically captures the temporal dynamics andcorrelations between features that best explain the target time series’ current behavior.
Revised: Our method corresponds to the maximum a posteriori estimation (MAP) in the ABP framework. Appendix A.2 formally presents the ABP algorithm and the connections to our method. To solve the reconstruction loss minimization problem, we use gradient-based methods. In particular, we rely on gradient descent (GD), randomly initializing the latent vector with independent and identically distributed Gaussian distributions and fixing the learning rate and the number of iterations as hyperparameters.

Original: Figure 2 demonstrates how SpectraNet ’s output evolves during the inference of z , adapting to current behaviours on the reference window.
Revised: Figure 2 demonstrates how SpectraNet ’s output evolves during the inference of z , adapting to current behaviors on the reference window.

Original: This optimization problem is non-convex, as the reconstruction follows equations 4 and 5.
Revised: Section 3.4 presents how SpectraNet is trained by alternating the inference and parameter learning steps. [Figure] The main advantage of inferring the latent vector is the robustness to handle both distribution shifts and missing values.

Original: However, given that all operations are differentiable, we can compute the gradient of the objective function w.r.t to z , allowing the use of gradient-based methods.
Revised: During this step, the model will infer the latent vector that best fits the current dynamics on the reference window, even if they follow an unseen behavior during training. The ability to dynamically adjust the temporal embedding for each window gives SpectraNet more robustness to changes in distribution, as shown in Figure 3. To deal with missing data, the reconstruction loss is masked. Following the notation in (Tashiro et al., 2021) , let M ∈ { 0 , 1 } M × T be the observation mask which indicates the availability of data.

Original: In particular, we rely on gradient descent (GD), randomly initializing the latent vector with independent and identically distributed Gaussian distributions and fixing the learning rate and the number of iterations as hyperparameters.
Revised: The latent vectors are inferred following [Equation] (6) where ◦ is the element-wise matrix multiplication. As demonstrated in the experimental section results on Tables 1, 2 and 4, and in congruity with previous studies such as (Han et al., 2017; Pang et al., 2020), inferring latent vectors provides superior robustness to missing data.

Original: 3. L ATENT S PACE S PECTRAL D ECOMPOSITION
Revised: 3. L ATENT S PACE S PECTRAL D ECOMPOSITION

Original: The second component of SpectraNet is the mapping from the latent vectors z into the temporal embedding E .
Revised: The second component of SpectraNet is the mapping from the latent vectors z into the temporal embedding E .

Original: Each element ofthe latent vector z corresponds to the coefficient of one element of 0.5 1.0 y
Revised: Each element of z corresponds to the coefficient of one element of the temporal basis

Original: SpectraNet output True

Original: End of reference window      120 1.0 0.5 0.

Original: y      120 1.0 0.5 0.

Original: [-y      
 Timestamp 1.0 0. [Figure] the temporal basis-] B ∈ R d × d t , where d is the number of elements and d t is the temporal length.
Revised: [-y      
 Timestamp 1.0 0. [Figure] the temporal basis-] B ∈ R d × d t , where d is the number of elements and d t is the temporal length.

Original: The i -th row of E is given by,
Revised: The i -th row of E is given by,

Original: [Equation]
Revised: [Equation]

Original: Each element of the basis consists of a predefined template function.
Revised: Each element of the basis consists of a predefined template function.

Original: Similarly to the N-BEATS model (Oreshkin et al., 2019), we split the basis into two types of patterns commonly found in time series: trends, represented by polynomial functions, and seasonal, represented by harmonic functions.
Revised: Similarly to previous work including (Oreshkin et al., 2019; Engel et al., 2020; Shan et al., 2022), the basis includes patterns commonly found in time series: trends, represented by polynomial functions, and seasonal, represented by harmonic functions.

Original: The final basis matrix B is the row-wise concatenation of the three following matrices.
Revised: The final basis matrix B is the row-wise concatenation of the three following matrices.

Original: [Equation]
Revised: [Equation]

Revised: where p is a hyperparameter that controls the max degree of the polynomial basis.

Original: The  temporal embedding E corresponds to a latent space spectral decomposition that encodes shared temporal dynamics of all features in the target window, as the latent vector z ∗ selects the relevant trend and frequency bands.
Revised: The final size of the basis d is equal to s w + p +1 . The temporal embedding E corresponds to a latent space spectral decomposition that encodes shared temporal dynamics of all features in the target window, as the latent vector z ∗ selects the relevant trend and frequency bands.

Original: Another crucial reason behind using a predefined basis is to impose strict temporal dependencies between the reference and forecasting windows.
Revised: Another crucial reason behind using a predefined basis is to impose strict temporal dependencies between the reference and forecasting windows.

Original: While inferring the latent vector, the forecasting window does not provide information (gradients).
Revised: While inferring the latent vector, the forecasting window does not provide information (gradients).

Original: If all the elements of E are  inferred(without a basis), given that we use a CNN with short kernels, the last values of the temporal embedding that determines the forecasting window cannot be optimized.
Revised: If all the entries of E are freely inferred the last values of the temporal embedding that determines the forecasting window cannot be optimized.

Original: 3. T OP -D OWN C ONVOLUTION N ETWORK
Revised: 3. T OP -D OWN C ONVOLUTION N ETWORK

Original: The last component of the model’s architecture is a top-down CNN, which produces the final forecast and reconstruction of the reference windowfor the M features simultaneously, ˆ Y t − L : t + H, from the temporal embedding E .
Revised: SpectraNet uses a Top-Down Convolution Network (CNN) as decoder, which produces the final forecast and reconstruction of the reference window ˆ Y t − L : t + H from the temporal embedding E .

Original: The full CNN architecture is presented in Figure 1, and can be formalized as:
Revised: The full CNN architecture is presented in Figure 1, and can be formalized as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Figure] where ConvTranspose1d is a transpose convolutional layer on the temporal dimension, BN is a batch normalization layer, and ReLU activations introduce non-linearity.
Revised: where Transposed Convolution is a transpose convolutional layer on the temporal dimension, BN is a batch normalization layer, and ReLU activations introduce non-linearity.

Original: The number of filters at each layer, kernel size, and stride are given in Appendix A.4.
Revised: The number of filters at each layer, kernel size, and stride are given in Appendix A.5.

Revised: The convolutional filters are not causal as in a Temporal Convolution Network (TCN) (Bai et al., 2018).

Revised: Instead, SpectraNet ’s causality , that allows the model to forecast future data using past observations, comes from inferring the latent vector z ∗ with the reference window.

Original: The firsttwo layers  learn a common representation for the window from the temporal embedding E .
Revised: The first layers of the CNN learn a common representation for all features from the temporal embedding E .

Original: The second layer refines the temporal resolution to the final size of the window s w .
Revised: Additionally, the second layer refines the temporal resolution to the final size of the window s w .

Original: The last layer produces the final output for all M features from h 2 , with a single convolutional filter for each feature.
Revised: The last layer produces the final output for all M features from h 2 .

Original: Note that the convolutional filters are not causal as in a TCN, meaning that the values for a temporal position of h and ˆ Y also depend on the next immediate temporal values (within the kernel size) from the previous layer.

Original: The causality of SpectraNet that allows the model to forecast from previous values is given by the fact that the latent vector z ∗ is only inferred with information on the reference window.
Revised: While equation 9 presents the default configuration for SpectraNet used in the experiments, additional layers can be added to increase the expressivity. Appendix A.10 presents ablation studies on the number of layers.

Original: 3. T RAINING PROCEDURE
Revised: 3. T RAINING PROCEDURE

Original: Each training iteration consists of two steps: the inference step and the learning step .
Revised: Each training iteration consists of two steps: the inference step and the learning step .

Original: During the inference step , the optimal latent vector z ∗ is inferred using Gradient Descent, solving the optimization problem given in equation 3and with the current parameters Θ .
Revised: During the inference step , the optimal latent vector z ∗ is inferred using Gradient Descent, solving the optimization problem given in equation 3 with the current parameters Θ .

Original: During the learning step , the latent vector is used as the input to the model, and the parameters Θ are updated using ADAM optimizer (Kingma & Ba, 2014) and MSE loss.
Revised: During the learning step , the latent vector is used as the input to the model, and the parameters Θ are updated using ADAM optimizer (Kingma & Ba, 2014) and MSE loss.

Original: For each iteration, we sample a small batch (with replacement) of multivariate time-series windows Y t : t + s w from the training data, each starting at a random timestamp t .
Revised: For each iteration, we sample a small batch (with replacement) of multivariate time-series windows Y t : t + s w from the training data, each starting at a random timestamp t .

Original: Each sampled observation is first normalized with the mean and standard deviation on the reference window, to decouple the scale and patterns (the output is scaled back before evaluation).
Revised: Each sampled observation is first normalized with the mean and standard deviation on the reference window, to decouple the scale and patterns (the output is scaled back before evaluation).

Original: The full training procedure is presented in Appendix A.5.
Revised: The full training procedure is presented in Appendix A.6.

Original: One potential drawback of inferring the latent vectorsinstead of using an encoder is the computational cost.
Revised: One potential drawback of inferring the latent vectors is the computational cost.

Original: We tackle this in several ways.
Revised: We tackle this in several ways.

Original: First, by solving the optimization problem using Gradient Descent, we can rely on automatic differentiation libraries such as PyTorch to efficiently compute the gradients.
Revised: First, by solving the optimization problem using GD, we can rely on automatic differentiation libraries such as PyTorch to efficiently compute the gradients.

Original: Second, backpropagation on CNN is parallelizable in GPU since it does not require sequential computation.
Revised: Second, backpropagation on CNN is parallelizable in GPU since it does not require sequential computation.

Original: The forecasts are independent between windows, so they can be computed in parallel, unlike RNN models, which must produce the forecasts sequentially.
Revised: The forecasts are independent between windows so they can be computed in parallel, unlike RNN models, which must produce the forecasts sequentially.

Original: Finally, during training, we persist the optimal latent vectors to future iterations on the same window.
Revised: Finally, during training, we persist the optimal latent vectors to future iterations on the same window.

Original: For a given observation t , the final latent vector z ∗ t is stored and used as a warm start when observation t is sampled again. This considerably reduces the number of iterations during the inference step .
Revised: For a given observation t , the final latent vector z ∗ t is stored and used as a warm start when observation t is sampled again, considerably reducing the number of iterations during the inference step .

Original: We discuss training times and memory complexity in section 4.
Revised: We discuss training times and memory complexity in section 4.

Original: 3. R OBUSTNESS TO MISSING VALUES AND DISTRIBUTIONAL SHIFTS
Revised: 3. R OBUSTNESS TO MISSING VALUES AND DISTRIBUTIONAL SHIFTS

Original: The inference of latent vectors is the primary driver of the superior robustness of SpectraNet to handle both distributional changes in the temporal dynamics and missing values.

Original: During this step, our model will infer the latent vectors that best describe the dynamics on the reference window, even if they follow a behavior never seen on the training data.

Original: SpectraNet can, therefore, dynamically adjust the temporal embedding to capture the current behavior on the reference window, giving more flexibility to extrapolate.

Original: To deal with missing values, we change the optimization on the reference window to only include the available information.

Original: SpectraNet is intrinsically robust to missing data; it does not require any adaptation to the architecture, training procedure, or interpolation models to fill the missing gaps.

Original: Our approach is even robust to the case where the missing data regime (how much or for how long missing data occurs) differs between the training and test sets.

Original: On the contrary, current SotA models require either modifications to the architecture or an interpolation model to first impute the missing values.

Original: Another advantage of our method is that it cannot only forecast directly on the raw data but by generating the entire window, it is simultaneously interpolating the past missing values and forecasting the future values.

Original: We first test our previous claims with a controlled experiment on synthetic data, which we refer to as Simulated7 .
Revised: We first test SpectraNet ’s robustness on synthetic data, which we name Simulated7 .

Original: We first generate seven time series of length 20,000, each generated as the sum of two cosines with random frequencies and small Gaussian noise.
Revised: This dataset consists of seven time series of length 20,000, each generated as the sum of two cosines with random frequencies and small Gaussian noise.

Original: Wethen inject missing values on both train and test sets, following the procedure described in section 4.
Revised: We inject missing values on both train and test sets, following the procedure described in section 4.

Original: For the distribution shifts experiments, we modifyonly the test set with two common distribution shifts: changes in trend and in magnitude .
Revised: For the distribution shifts experiments, we modify the test set with two common distribution shifts: changes in trend and in magnitude .

Original: Figure 3 presents the forecasts for SpectraNet for the three settings considered.
Revised: Figure 3 presents the forecasts for SpectraNet for the three settings considered.

Original: Weonly include the first feature in this figure, and we show the complete data in Appendix A.6.
Revised: We include the first feature in this figure, and we show the complete data in Appendix A.7.

Original: As seen in panel (a), our method can accurately forecast the ground truth even with 80% of missing values, a challenging setting since  intervals where consecutive timestamps are available are short.
Revised: As seen in panel (a), our method can accurately forecast the ground truth even with 80% of missing values, a challenging setting since [Figure] intervals where consecutive timestamps are available are short.

Original: This setting also demonstrates how SpectraNet can accurately learn correlations between features, as in many reference windows only few features contain information.
Revised: This setting also demonstrates how SpectraNet can accurately learn correlations between features, as in many reference windows only few features contain information.

Original: Panels (b) and (c) demonstrate how SpectraNet can dynamically adapt to new data behaviors, even if the model never observed them during training. 
Revised: Panels (b) and (c) demonstrate how SpectraNet can dynamically adapt to new data behaviors, even if the model never observed them during training. Table 1 presents a quantitative comparison. [Table] [Figure]

Revised: [Table]

Original: Table 1 presents a quantitative comparison between the performance of  SpectraNet and baseline models
Revised: [Figure] [Table] In Appendix A.9 we demonstrate how SpectraNet is also robust to changes over time of the missing data regime (how much or for how long missing data occurs). While SpectraNet is intrinsically robust to missing data

Original: [Table]
Revised: [Table]

Revised: current SoTA models require either modifications to the architecture or an interpolation model to first impute the missing values.

Revised: Another advantage of our method is that it cannot only forecast directly on the raw data

Revised: [Table]

Revised: but by generating the entire window

Revised: [Table]

Revised: it is simultaneously interpolating the past missing data and forecasting the future values

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We base our experimental setting, benchmark datasets, train/validation/test splits, hyperparameter tuning, and data processing on previous works on multivariate forecasting (Zhou et al., 2021; Wu et al., 2021; Challu et al., 2022a).
Revised: We base our experimental setting, benchmark datasets, train/validation/test splits, hyperparameter tuning [Table] and data processing on previous works on multivariate forecasting (Zhou et al., 2021; Wu et al., 2021; Challu et al., 2022a).

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We evaluate our model on synthetic data and five popular benchmark datasets commonly used in the forecasting literature, comprising various applications and domains.
Revised: We evaluate our model on synthetic data and five popular benchmark datasets commonly used in the forecasting literature, comprising various applications and domains.

Original: All datasets are normalized with the mean and standard deviation computed on the train set.
Revised: All datasets are normalized with the mean and standard deviation computed on the train set.

Original: We split each dataset into train, validation, and test sets, following different proportions based on the number of timestamps.
Revised: We split each dataset into train, validation, and test sets, following different proportions based on the number of timestamps.

Original: We present summary statistics for each dataset in Appendix A.2.
Revised: We present summary statistics for each dataset in Appendix A.3.

Original: The Influenza-like illness (ILI) dataset contains the weekly proportion of patients with influenzalike symptoms in the US, reported by the Centers for Disease Control and Prevention, from 2002 to 2021.
Revised: The Influenza-like illness (ILI) dataset contains the weekly proportion of patients with influenzalike symptoms in the US, reported by the Centers for Disease Control and Prevention, from 2002 to 2021.

Original: Exchange reports the daily exchange rates of eight currencies relative to the US dollar from 1990 to 2016.
Revised: Exchange reports the daily exchange rates of eight currencies relative to the US dollar from 1990 to 2016.

Original: The Solar dataset contains the hourly photo-voltaic production of 32 solar stations in Wisconsin during 2016.
Revised: The Solar dataset contains the hourly photo-voltaic production of 32 solar stations in Wisconsin during 2016.

Original: Finally, the Electricity Transformer Temperature (ETTm2) dataset is a stream of eight sensor measurements of an electricity transformer in China from July 2016 to
Revised: Finally, the Electricity Transformer Temperature (ETTm2) dataset is a stream of eight sensor measurements of an electricity transformer in China from July 2016 to

Original: July 2018.
Revised: July 2018.

Original: Weather contains 21 meteorological conditions recorded at the German Max Planck
Revised: Weather contains 21 meteorological conditions recorded at the German Max Planck

Original: Biogeochemistry Institute in 2021.
Revised: Biogeochemistry Institute in 2021.

Original: 4. T RAINING AND E VALUATION SETUP
Revised: 4. T RAINING AND E VALUATION SETUP

Original: We train models on the training set comprising the earliest history of each dataset.
Revised: We train models on the training set comprising the earliest history of each dataset.

Original: We select the optimal hyperparameters for all models (including the baselines to ensure fair comparisons) on each dataset and occlusion percentage based on the performance (measured by MSE) on the validation set, using a Bayesian optimization algorithm, HYPEROPT (Bergstra et al., 2011).
Revised: We select the optimal hyperparameters for all models (including the baselines to ensure fair comparisons) on each dataset and occlusion percentage based on the performance (measured by MSE) on the validation set, using a Bayesian optimization algorithm, HYPEROPT (Bergstra et al., 2011).

Original: For SpectraNet we tune the number of convolutional filters, temporal size of the latent vectors, window normalization, and optimization hyperparameters for both the inferencial and learning steps.
Revised: For SpectraNet we tune the number of convolutional filters, temporal size of the latent vectors, window normalization, and optimization hyperparameters for both the inferencial and learning steps.

Original: The complete list of hyperparameters is included in Appendix A.4.
Revised: The complete list of hyperparameters is included in Appendix A.5.

Original: Finally, we use the model with the optimal configuration to forecast the test set.

Original: For the main results, we use a multi-step forecast with a horizon of size 24, and forecasts are produced in a rolling window strategy.
Revised: For the main results, we use a multi-step forecast with a horizon of size 24, and forecasts are produced in a rolling window strategy.

Original: We repeat the experiment five times with different random seeds and report average performances.
Revised: We repeat the experiment five times with different random seeds and report average performances.

Original: We run the experiments on an AWS g3.4xlarge EC2 instance with NVIDIA Tesla M60 GPUs.
Revised: We run the experiments on an AWS g3.4xlarge EC2 instance with NVIDIA Tesla M60 GPUs.

Original: We compare our proposed model against several univariate and multivariate state-of-the-art baselines based on different architectures: Transformers, feed-forward networks (MLP), Graph Neural
Revised: We compare our proposed model against several univariate and multivariate state-of-the-art baselines based on different architectures: Transformers, feed-forward networks (MLP), Graph Neural

Original: Network (GNN), and Recurrent Neural Network (RNN).
Revised: Network (GNN), and Recurrent Neural Network (RNN).

Original: For Transformer-based models we include the Informer (Zhou et al., 2021) and Autoformer (Wu et al., 2021), two recent models for multi-step long-horizon forecasting; for MLP we consider the univariate N-HiTS (Challu et al., 2022a) and
Revised: For Transformer-based models we include the Informer (Zhou et al., 2021) and Autoformer (Wu et al., 2021), two recent models for multi-step long-horizon forecasting; for MLP we consider the univariate N-HiTS (Challu et al., 2022a) and

Original: N-BEATS (Oreshkin et al., 2019) models; for GNN models we include the StemGNN (Cao et al., 2020), and lastly we include a multi-step univariate RNN with dilations (Chang et al., 2017).
Revised: N-BEATS (Oreshkin et al., 2019) models; for GNN models we include the StemGNN (Cao et al., 2020), and lastly we include a multi-step univariate RNN with dilations (Chang et al., 2017).

Original: 4. M ISSING DATA SETUP
Revised: 4. M ISSING DATA SETUP

Original: We present a new experimental setting for testing the forecasting models’ robustness to missing data.
Revised: We present a new experimental setting for testing the forecasting models’ robustness to missing data.

Original: We parameterize the experiments with two parameters: the size of missing segments s and the probability that each segment is missing with p o .
Revised: We parameterize the experiments with two parameters: the size of missing segments s and the probability that each segment is missing with p o .

Original: First, the original time series is divided into disjoint segments of length s .
Revised: First, the original time series is divided into disjoint segments of length s .

Original: Second, each feature m in each segment is occluded with probability p o .
Revised: Second, each feature m in each segment is occluded with probability p o .

Original: We repeat the experiment with different probabilities p o : 0% (no missing values), 20%, 40%, 60%, and 80%, to test models’ performance under different proportions of missing values.
Revised: We repeat the experiment with different probabilities p o : 0% (no missing values), 20%, 40%, 60%, and 80%, to test models’ performance under different proportions of missing values.

Original: The size of segments s is fixed at ten timestamps for ILI and at 100 for all other datasets.
Revised: The size of segments s is fixed at ten timestamps for ILI and at 100 for all other datasets.

Original: Figure 3 shows an example of the setting for the Simulated7 dataset with 80% missing data (occluded timestamps in grey).
Revised: Figure 3 shows an example of the setting for the Simulated7 dataset with 80% missing data (occluded data in grey).

Original: All models are trained with the training loss only on available values since we observed this improves the forecasting accuracy.

Original: 4. K EY RESULTS
Revised: 4. K EY RESULTS

Original: Forecasting accuracy .
Revised: Forecasting accuracy .

Original: Table 2 presents the forecasting accuracy.
Revised: Table 2 presents the forecasting accuracy.

Original: First, SpectraNet achieved
Revised: First, SpectraNet achieved

Original: SoTA performance even on full data, showingthat the advantages of our method are not limited to the robustness to missing data and distribution shifts.
Revised: SoTA performance even on full data, showing the advantages of our method are not limited to the robustness to missing data and distribution shifts.

Original: SpectraNet outperformed N-HiTS/NBEATS in two of the five datasets and placed in the Top-3 models across all datasets, outperforming
Revised: SpectraNet outperformed N-HiTS/NBEATS in two of the five datasets and placed in the Top-3 models across all datasets, outperforming

Original: Transformer-based models, StemGNN, and DilRNN.
Revised: Transformer-based models, StemGNN, and DilRNN.

Original: SpectraNet also achieves the best performance among all multivariate models.
Revised: SpectraNet also achieves the best performance among all multivariate models.

Original: The superior robustness of SpectraNet to missing values is evident in all datasets and proportions.
Revised: The superior robustness of SpectraNet to missing values is evident in all datasets and proportions.

Original: In datasets with strong seasonalities, such as Simulated7 and Solar , the accuracy of
Revised: In datasets with strong seasonalities, such as Simulated7 and Solar , the accuracy of

Original:  SpectraNet marginally degrades with even 80% of missing values.
Revised: [Table] SpectraNet marginally degrades with even 80% of missing values.

Original: The relative performance of our method improves with the proportion of missing data.
Revised: The relative performance of our method improves with the proportion of missing data.

Original: For example, for 80%, the average MSE[Table] across datasets is 48% lower than the N-HiTS and 65% lower than the Autoformer.
Revised: For example, for 80%, the average MSE across datasets is 48% lower than the N-HiTS and 65% lower than the Autoformer.

Original: The improvement on ILI with complete data of almost 50% against the second best model is explained by the distribution shift between the train and test set [Table] where the latter presents larger spikes and a clear positive trend on some features.
Revised: The improvement on ILI with complete data of almost 50% against the second best model is explained by the distribution shift between the train and test set, where the latter presents larger spikes and a clear positive trend on some features.

Original: Appendix A.7 presents a plot of the ILI dataset.
Revised: Appendix A.8 presents a plot of the ILI dataset.

Original: Imputation accuracy .
Revised: Imputation accuracy .

Original: Table 3 presents the results for the imputation task.
Revised: Table 3 presents the results for the imputation task.

Original: We include a SoTA imputation diffusion model,
Revised: We include a SoTA imputation diffusion model,

Original: CSDI
Revised: CSDI

Original: (Tashiro et al., 2021) [Table] and three simple baselines: (i) imputation with the mean of each feature, (ii) imputation with the last available value (Naive), and (iii) linear interpolation between past and future available values.
Revised: (Tashiro et al., 2021), and three simple baselines: (i) imputation with the mean of each feature, (ii) imputation with the last available value (Naive), and (iii) linear interpolation between past and future available values.

Original: SpectraNet consistently achieves the best performance across all datasets.
Revised: SpectraNet consistently achieves the best performance across all datasets.

Original: CSDI outperformes all simpler baselines on Simulated7 and Solar, but its performance degrades on datasets with distribution shifts such as Simulated7-Trend and ILI.
Revised: CSDI outperformes all simpler baselines on Simulated7 and Solar, but its performance degrades on datasets with distribution shifts such as Simulated7-Trend and ILI.

Original: [Table]
Revised: [Table]

Original: Memory and time complexity .
Revised: Memory and time complexity .

Original: We compare the training time and memory usage (as the number of learnable parameters) of SpectraNet and baseline models as a function of the input size on the ETTm2 dataset in Figure 4, using the optimal hyperparameter configuration.
Revised: We compare the training time and memory usage (as the number of learnable parameters) of SpectraNet and baseline models as a function of the input size on the ETTm2 dataset in Figure [Table] using the optimal hyperparameters.

Original: Panel (a) shows our method has  the lowest memory footprint, with up to 85% fewer parameters than the N-HiTS (the second best alternative in most datasets) and 92% than the Autoformer.
Revised: Panel (a) shows our method has [Figure] the lowest memory footprint, with up to 85% fewer parameters than the N-HiTS (the second best alternative in most datasets) and 92% than the Autoformer.

Original: Despite the fact that 
Revised: Despite the fact that SpectraNet performs the inference step in each iteration

Original: [Figure]
Revised: [Figure]

Original: SpectraNet performs the inference step in each iteration, thanks to the improvements discussed in section 3 and the reduced number of parameters, training times are comparable to other baseline models.
Revised: thanks to the improvements discussed in section 3 and the reduced number of parameters, training times are comparable to other baseline models.

Original: [Figure]
Revised: [Figure]

Original:  Finally, we test the contribution of the two major components proposed in this work, the dynamic inference of latent vectors and the LSSD.
Revised: Ablation studies . Finally, we test the contribution of the two major components proposed in this work, the dynamic inference of latent vectors and the LSSD.

Original: We compare SpectraNet against two versions without these components.
Revised: We compare SpectraNet against two versions without these components.

Original: In SpectraNet 1 we add a CNN encoder to map inputs to a standard embedding, and in SpectraNet 2 we keep the inference step but remove the LSSD.
Revised: In SpectraNet 1 we add a CNN encoder to map inputs to a standard embedding, and in SpectraNet 2 we keep the inference step but remove the LSSD.

Original: Table 4 presents the average performance across the five benchmark datasets.
Revised: Table 4 presents the average performance across the five benchmark datasets.

Original: As expected,
Revised: As expected,

Original: SpectraNet 1 with the parametric encoder is not robust to missing data, with similar performance to other
Revised: SpectraNet 1 with the parametric encoder is not robust to missing data, with similar performance to other

Original: SoTA models.
Revised: SoTA models.

Original: On top of improving the performance over
Revised: On top of improving the performance over

Original: SpectraNet 2 , the LSSD reduces the number of parameters by 70% since fewer layers are needed to generate the complete window.
Revised: SpectraNet 2 , the LSSD reduces the number of parameters by 70% since fewer layers are needed to generate the complete window.

Original:  D ISCUSSIONS AND C ONCLUSION
Revised:  D ISCUSSIONS AND C ONCLUSION

Original: This work proposes a novel multivariate time-series forecasting model that uses a Top-Down CNN to generate time-series windows from a novel latent space spectral decomposition.
Revised: This work proposes a novel multivariate time-series forecasting model that uses a Top-Down CNN to generate time-series windows from a novel latent space spectral decomposition.

Original: It dynamically infers the latent vectors that best explain the current dynamics, considerably improving the robustness to distribution shifts and missing data.
Revised: It dynamically infers the latent vectors that best explain the current dynamics, considerably improving the robustness to distribution shifts and missing data.

Original: We compare the accuracy of our method with
Revised: We compare the accuracy of our method with

Original: SoTA models based on several architectures for forecasting and interpolation tasks.
Revised: SoTA models based on several architectures for forecasting and interpolation tasks.

Original: We demonstrate
Revised: We demonstrate

Original: SpectraNet does not only achieve SoTA on benchmark datasets but can also produce accurate forecasts under some forms of distribution shifts and extreme cases of missing data.
Revised: SpectraNet does not only achieve SoTA on benchmark datasets but can also produce accurate forecasts under some forms of distribution shifts and extreme cases of missing data.

Original: Our experiments provide evidence that, as expected, the performance of current SoTA forecasting models significantly degrades under the presence of missing data and distribution shifts.
Revised: Our experiments provide evidence that, as expected, the performance of current SoTA forecasting models significantly degrades under the presence of missing data and distribution shifts.

Original: These challenges are commonly present in high stake settings such as healthcare (for instance, cases with up to 80% missing data are common) and are becoming more predominant in many domains with the recent global events such as the COVID-19 pandemic.
Revised: These challenges are commonly present in high stake settings such as healthcare (for instance, cases with up to 80% missing data are common) and are becoming more predominant in many domains with the recent global events such as the COVID-19 pandemic.

Original: Designing robust algorithms and solutions that tackle these challenges can significantly improve their adoption and increase their benefits.
Revised: Designing robust algorithms and solutions that tackle these challenges can significantly improve their adoption and increase their benefits.

Original: We believe thatthe technique of dynamically inferring latent vectors can have multiple other applications in time-series forecasting.
Revised: We believe that dynamically inferring latent vectors can have multiple other applications in timeseries forecasting.

Original: For instance, it can have applications in transfer learning and few-shot learning.
Revised: For instance, it can have applications in transfer learning and few-shot learning.

Original: A pre-trained CNN (or any decoder) could forecast completely unseen time series, while the inference step will allow the model to adapt to different temporal patterns.
Revised: A pre-trained decoder could forecast completely unseen time series, while the inference step will allow the model to adapt to different temporal patterns.

Original: Moreover, the inference step can be adapted to produce multiple samples and calibrated to learn the target time series distribution, allowing the model to produce probabilistic forecasts.
Revised: Moreover, the inference step can be adapted to produce multiple samples and calibrated to learn the target time series distribution.

Original: A A PPENDIX A. R ELATED W ORK
Revised: A A PPENDIX A. R ELATED W ORK

Original: Time-series forecasting.
Revised: Time-series forecasting.

Original: Time-series forecasting has long been an active area of research in academia and industry due to its broad range of high-impact applications.
Revised: Time-series forecasting has long been an active area of research in academia and industry due to its broad range of high-impact applications.

Original: The latest developments in machine learning have been combined with classical methods or used to develop new models (Januschowski et al., 2020) which have been successful in recent large-scale forecasting competitions (Makridakis et al., 2020; 2022).
Revised: The latest developments in machine learning have been combined with classical methods or used to develop new models (Januschowski et al., 2020) which have been successful in recent large-scale forecasting competitions (Makridakis et al., 2020; 2022).

Original: The earlier breakthroughs in modeling time-series with Deeplearning (Januschowski et al., 2018) include the widely used Recurrent Neural Networks (RNN) (Chang et al., 2017; Salinas et al., 2020; 2019) and Temporal Convolution Networks (TCN) (Bai et al., 2018).
Revised: The earlier breakthroughs in modeling time-series with Deeplearning (Januschowski et al., 2018) include the widely used Recurrent Neural Networks (RNN) (Chang et al., 2017; Salinas et al., 2020; 2019) and Temporal Convolution Networks (TCN) (Bai et al., 2018).

Original: The success of Transformers (Vaswani et al., 2017) in sequential data, such as natural language processing (NLP) and audio processing, inspired many recent models with attention mechanisms.
Revised: The success of Transformers (Vaswani et al., 2017) in sequential data, such as natural language processing (NLP) and audio processing, inspired many recent models with attention mechanisms.

Original: The Informer (Zhou et al., 2021) introduces a Prob-sparse self-attention to reduce the quadratic complexity of vanilla Transformers; the Autoformer (Wu et al., 2021) proposed a decomposition architecture in trend and seasonal components, and the Auto-correlation mechanism.
Revised: The Informer (Zhou et al., 2021) introduces a Prob-sparse self-attention to reduce the quadratic complexity of vanilla Transformers; the Autoformer (Wu et al., 2021) proposed a decomposition architecture in trend and seasonal components, and the Auto-correlation mechanism.

Original: Simpler approaches based on feed-forward networks have also shown remarkable performance.
Revised: Simpler approaches based on feed-forward networks have also shown remarkable performance.

Original: The
Revised: The

Original: N-BEATS model (Oreshkin et al., 2019) uses a deep stack of fully-connected layers to decompose the forecast into both learnable and predefined basis functions; the NBEATSx extension incorporates interpretable exogenous variable decomposition (Olivares et al., 2022); the N-HiTS model (Challu et al., 2022a) generalizes the N-BEATS by introducing mixed-data sampling and hierarchial interpolation to decompose the forecast in different frequencies.
Revised: N-BEATS model (Oreshkin et al., 2019) uses a deep stack of fully-connected layers to decompose the forecast into both learnable and predefined basis functions; the NBEATSx extension incorporates interpretable exogenous variable decomposition (Olivares et al., 2022); the N-HiTS model (Challu et al., 2022a) generalizes the N-BEATS by introducing mixed-data sampling and hierarchial interpolation to decompose the forecast in different frequencies.

Original: Finally, Graph Neural Networks (GNN) has been used to incorporate complex relations between a large number of time series, including the
Revised: Finally, Graph Neural Networks (GNN) has been used to incorporate complex relations between a large number of time series, including the

Original: GraphWaveNet (Wu et al., 2019), and StemGNN (Cao et al., 2020) models.
Revised: GraphWaveNet (Wu et al., 2019), and StemGNN (Cao et al., 2020) models.

Original: Time-series imputation.
Revised: Time-series imputation.

Original: The standard practice to handle missing data is filling the missing information, a process called interpolation .
Revised: The standard practice to handle missing data is filling the missing information, a process called interpolation .

Original: Simple interpolation alternatives include replacing missing values with zeros, the mean, most-recent value (naive), and linear interpolation.
Revised: Simple interpolation alternatives include replacing missing values with zeros, the mean, most-recent value (naive), and linear interpolation.

Original: Most recent Deeplearning approaches consist of Generative Adversarial Networks (GANs) and RNN-based architectures.
Revised: Most recent Deeplearning approaches consist of Generative Adversarial Networks (GANs) and RNN-based architectures.

Original: Some notable examples are E2gan (Luo et al., 2019),
Revised: Some notable examples are E2gan (Luo et al., 2019),

Original: Brits (Cao et al., 2018), and NAOMI (Liu et al., 2019).
Revised: Brits (Cao et al., 2018), and NAOMI (Liu et al., 2019).

Original: More recent approaches include the CDSI (Tashiro et al., 2021) model, a score-based diffusion auto-regressive architecture that produces a distribution for the imputed values.
Revised: More recent approaches include the CDSI (Tashiro et al., 2021) model, a score-based diffusion auto-regressive architecture that produces a distribution for the imputed values.

Original: Alternating back-propagation .
Revised: Alternating back-propagation .

Original: The method we propose to infer latent vectors is inspired by the
Revised: The method we propose to infer latent vectors is inspired by the

Original: Alternating back-propagation algorithm (ABP) for Generative models (Han et al., 2017).
Revised: Alternating back-propagation algorithm (ABP) for Generative models (Han et al., 2017).

Original: The key idea of this algorithm is to sample latent vectors from the posterior distribution with MCMC methods and train a Generative model by maximizing the observed likelihood directly.
Revised: The key idea of this algorithm is to sample latent vectors from the posterior distribution with MCMC methods and train a Generative model by maximizing the observed likelihood directly.

Original: Generative models trained with ABP do not need an encoder, such as Variational Autoencoders (VAE), or Discriminator networks, such as GANs.
Revised: Generative models trained with ABP do not need an encoder, such as Variational Autoencoders (VAE), or Discriminator networks, such as GANs.

Original: More recent work extend the original architecture with energy-based models for CV (Pang et al., 2020), LSTM networks for text generation (Pang et al., 2021), and a hierarchical latent space for anomaly detection (Challu et al., 2022b).
Revised: More recent work extend the original architecture with energy-based models for CV (Pang et al., 2020), LSTM networks for text generation (Pang et al., 2021), and a hierarchical latent space for anomaly detection (Challu et al., 2022b).

Original: A. B ENCHMARK DATASETS
Revised: A. A LTERNATING BACK - PROPAGATION

Revised: This section presents the principles of Alternating back-propagation (ABP) to train a generative model, as proposed in (Han et al., 2017) and presented in (Challu et al., 2022b), and the connection with our proposed latent vector inference.

Revised: Let y ∈

Revised: R D be a D -dimension data vector (such as an image or time-series window) and f be a generative model with parameters θ ,

Revised: [Equation]

Revised: where

Revised: z ∈ R

Revised: d are latent factors and d < D .

Revised: Let { y ( i ) , i = 1 , ..., n } be n training observations.

Revised: In

Revised: Alternating Back-Propagation, parameters θ are trained by maximizing the observed log-likelihood

Revised: [Equation]

Revised: The gradients L ′ ( θ ) for observation i are given by,

Revised: [Equation]

Revised: (12) where p θ ( z ( i ) | y

Revised: ( i ) ) is the posterior distribution.

Revised: ABP approximates this intractable expectation with

Revised: Monte Carlo average by taking a single sample z ∗ from the posterior distribution with approximate

Revised: Langevin Dynamics, by iterating

Revised: [Equation]

Revised: (13) where ϵ t ∼ N (0 , I D ) , s is the step size, and σ z controls the annealing or tempering.

Revised: Finally, the

Revised: Monte Carlo approximation of the gradient is given by

Revised: [Equation]

Revised: For our proposed latent vector inference, we reformulate the posterior sampling as a non-convex minimization problem presented in 3, which aims to minimize the mean square error (MSE) between the reconstruction and ground truth y

Revised: ( i ) .

Revised: Minimizing the MSE is equivalent to the maximum a posteriori estimation (MAP) of p θ ( z ( i ) | y

Revised: ( i ) )

Revised: assuming Gaussian distributions as in equation 10.

Revised: This methodology also allows for using more sophisticated gradient-based methods than equation 13, such as Adam, can be used to improve the convergence.

Revised: A. B ENCHMARK DATASETS

Original: Table 5 presents summary information for the benchmark datasets, including the granularity (frequency), number of features and timestamps, and train/validation/test proportions.
Revised: Table 5 presents summary information for the benchmark datasets, including the granularity (frequency), number of features and timestamps, and train/validation/test proportions.

Original: [Table]
Revised: [Table]

Original: All datasets are public, and are available in the following links:
Revised: All datasets are public, and are available in the following links:

Original: • ILI : https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html • Exchange : https://github.com/laiguokun/multivariate-time-series-data •
Revised: • ILI : https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html • Exchange : https://github.com/laiguokun/multivariate-time-series-data •

Original: Solar : https://www.nrel.gov/grid/solar-power-data.html • ETTm2 : https://github.com/zhouhaoyi/ETDataset •
Revised: Solar : https://www.nrel.gov/grid/solar-power-data.html • ETTm2 : https://github.com/zhouhaoyi/ETDataset •

Original: Weather : https://www.bgc-jena.mpg.de/wetter/
Revised: Weather : https://www.bgc-jena.mpg.de/wetter/

Original: A
Revised: A

Original:  S IMULATED 7 DATASET
Revised:  S IMULATED 7 DATASET

Original: Simulated7 is a synthetic dataset that we design to evaluate forecasting models robustness to missing data and distribution shifts in a controlled environment.
Revised: Simulated7 is a synthetic dataset that we design to evaluate forecasting models robustness to missing data and distribution shifts in a controlled environment.

Original: It consists of 7 time-series, each generated independently as the sum of two cosine functions with different frequencies and a small
Revised: It consists of 7 time-series, each generated independently as the sum of two cosine functions with different frequencies and a small

Original: Gaussian noise.
Revised: Gaussian noise.

Original: In particular, it is composed as the sum of the three following elements:
Revised: In particular, it is composed as the sum of the three following elements:

Original: [Equation]
Revised: [Equation]

Original: Each time-series consists of 20,000 timestamps, with regular intervals between [0 , 5] .
Revised: Each time-series consists of 20,000 timestamps, with regular intervals between [0 , 5] .

Original: For the missing data we obfuscate random timestamps following the procedure described in section 4, using s = 10 and p o ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
Revised: For the missing data we obfuscate random timestamps following the procedure described in section 4, using s = 10 and p o ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .

Original: For simulating distribution shifts we perturb only the test set with two transformations: adding a linear trend with slope 6, and scaling the magnitude by 0.5.
Revised: For simulating distribution shifts we perturb only the test set with two transformations: adding a linear trend with slope 6, and scaling the magnitude by 0.5.

Original: A. H YPERPARAMETER OPTIMIZATION
Revised: A. H YPERPARAMETER OPTIMIZATION

Original: SpectraNet hyperparameters are tuned on the validation set of each dataset using HYPEROPT algorithm with 30 iterations, Table 6 presents the hyperparameter grid.
Revised: SpectraNet hyperparameters are tuned on the validation set of each dataset using HYPEROPT algorithm with 30 iterations, Table 6 presents the hyperparameter grid.

Original: To ensure a fair comparison with baseline models, we also tuned their respective hyperparameters with the same procedure, as the default configuration in their implementations might not perform well in the different settings we explore.
Revised: To ensure a fair comparison with baseline models, we also tuned their respective hyperparameters with the same procedure, as the default configuration in their implementations might not perform well in the different settings we explore.

Original: We detail the hyperparameter grid for each baseline model next.
Revised: We detail the hyperparameter grid for each baseline model next.

Original: [Table]
Revised: [Table]

Original: For N-HiTS, we use the hyperparameter grid described in their paper (Challu et al., 2022a).
Revised: For N-HiTS, we use the hyperparameter grid described in their paper (Challu et al., 2022a).

Original: The
Revised: The

Original: N-HiTS is a generalization of the N-BEATS, which is already included in the hyperparameter grid as a posible configuration.
Revised: N-HiTS is a generalization of the N-BEATS, which is already included in the hyperparameter grid as a posible configuration.

Original: For the Informer and Autoformer we explore different values for the dropout probability, number of heads, learning rate (same as SpectraNet ), batch size (same as
Revised: For the Informer and Autoformer we explore different values for the dropout probability, number of heads, learning rate (same as SpectraNet ), batch size (same as

Original: SpectraNet ), size of embedding, and sequence length.
Revised: SpectraNet ), size of embedding, and sequence length.

Original: For StemGNN we explore different optimization parameters, including the learning rate, batch size, and epochs.
Revised: For StemGNN we explore different optimization parameters, including the learning rate, batch size, and epochs.

Original: Finally, for RNN we explore different dilations, number of layers, hidden size, and optimization parameters.
Revised: Finally, for RNN we explore different dilations, number of layers, hidden size, and optimization parameters.

Original: For the NHiTS, transformers, and RNN models we used the Neuralforecast library (available in PyPI and Conda).
Revised: For the NHiTS, transformers, and RNN models we used the Neuralforecast library (available in PyPI and Conda).

Original: A. S PECTRA N ET TRAINING ALGORITHM
Revised: A. S PECTRA N ET TRAINING ALGORITHM

Original: Algorithm 1 presents the training procedure of SpectraNet .
Revised: Algorithm 1 presents the training procedure of SpectraNet .

Original: The model is trained for a fixed number of iterations n iters , randomly sampling b windows in each iteration.
Revised: The model is trained for a fixed number of iterations n iters , randomly sampling b windows in each iteration.

Original: Parameters θ are optimized with Adam optimizer.
Revised: Parameters θ are optimized with Adam optimizer.

Original: input : multivariate time-series Y ∈ R m × T , model F θ , learning iterations n iters output: F θ ∗ , inferred latent vectors { z ∗ t , t = 0 , ..., T }
Revised: input : multivariate time-series Y ∈ R m × T , model F θ , learning iterations n iters output: F θ ∗ , inferred latent vectors { z ∗ t , t = 0 , ..., T }

Original: Let i ← 0 , initialize θ
Revised: Let i ← 0 , initialize θ

Original: Initialize z t , for t = 0 , ..., T while i < n iters do
Revised: Initialize z t , for t = 0 , ..., T while i < n iters do

Original: Take a random mini-batch of windows { Y j k − L : j k + H , j k ∼ U ( L, T − H ) , k { 1 , ..., b } .
Revised: Take a random mini-batch of windows { Y j k − L : j k + H , j k ∼ U ( L, T − H ) , k { 1 , ..., b } .

Original: z ∗ j k ← arg min
Revised: z ∗ j k ← arg min

Original: z L ( Y j k − L : j k , ˆ Y j k − L : j k ( z )) , k ∈ { 1 , ..., b }
Revised: z L ( Y j k − L : j k , ˆ Y j k − L : j k ( z )) , k ∈ { 1 , ..., b }

Original: Update θ with Adam using z ∗ as input.
Revised: Update θ with Adam using z ∗ as input.

Original: Store z ∗ j k , k ∈ { 1 , ..., b } i ← i + 1 end
Revised: Store z ∗ j k , k ∈ { 1 , ..., b } i ← i + 1 end

Original: A. F ORECASTS ON S IMULATED
Revised: A. F ORECASTS ON S IMULATED

Original: Figure 1 presents the complete forecasts on the test set for SpectraNet , N-HiTS, Informer
Revised: Figure 1 presents the complete forecasts on the test set for SpectraNet , N-HiTS, Informer

Original: [Figure]
Revised: [Figure]

Original: and
Revised: and

Original: RNN models.
Revised: RNN models.

Original: SpectraNet is the only model that can accurately forecast with up to 80% missing data and changes in distribution.
Revised: SpectraNet is the only model that can accurately forecast with up to 80% missing data and changes in distribution.

Original: [Figure]
Revised: [Figure]

Original: Missing values 
Revised: Missing values - SpectraNet

Original: [Figure]
Revised: [Figure]

Original: SpectraNet

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (f) Magnitude - N-HiTS
Revised: (f) Magnitude - N-HiTS

Original: [Figure]
Revised: [Figure]

Original: TrueForecast
Revised: TrueForecast

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: h)
Revised: h)

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. ILI DATASET
Revised: A. ILI DATASET

Original: Figure 2 shows the time series of the ILI dataset and the start of the test set (vertical red line).
Revised: Figure 2 shows the time series of the ILI dataset and the start of the test set (vertical red line).

Original: The first five features present considerably larger and longer spikes on the test set.
Revised: The first five features present considerably larger and longer spikes on the test set.

Original: This is a particularly challenging task, as most models tend to underestimate the larger spikes.
Revised: This is a particularly challenging task, as most models tend to underestimate the larger spikes.

Original: [Figure]
Revised: [Figure]

Original: A. A DDITIONAL OCCLUSION EXPERIMENTS
Revised: A. A DDITIONAL OCCLUSION EXPERIMENTS

Original: In this section we present additional occlusion experiments to analyze the case where the missing data regime (how much or for how long missing data occurs) differs between the training and test sets.
Revised: In this section we present additional occlusion experiments to analyze the case where the missing data regime (how much or for how long missing data occurs) differs between the training and test sets.

Original: In this setting we train models on complete data and inject missing values only during inference on the test set.
Revised: In this setting we train models on complete data and inject missing values only during inference on the test set.

Original: The following table presents SpectraNet and N-HiTS (best baseline) results on the Simulated7 dataset.
Revised: The following table presents SpectraNet and N-HiTS (best baseline) results on the Simulated7 dataset.

Original: SpectraNet performance is almost identical to the original results, demonstrating the method’s robustness to changes in the behavior of the presence of missing data
Revised: SpectraNet performance is almost identical to the original results, demonstrating the method’s robustness to changes in the behavior of the presence of missing data

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: A.S PECTRA N ET CNN ABBLATION STUDIES
Revised: A. CNN ABLATION STUDIES

Original: Figure 1 of the paper presents the default recommended configuration for SpectraNet .
Revised: Figure 1 of the paper presents the default recommended configuration for SpectraNet .

Original: In this section we explore different configurations for the CNN, in particular for the number of hidden layers.
Revised: In this section we explore different configurations for the CNN, in particular for the number of hidden layers.

Original: For the model SpectraNet i,j , i refers to the number of layers with temporal resolution s w / 2 , and j to the number of layers with s w resolution.
Revised: For the model SpectraNet i,j , i refers to the number of layers with temporal resolution s w / 2 , and j to the number of layers with s w resolution.

Original: The following table presents the results on ILI and Solar datasets.
Revised: The following table presents the results on ILI and Solar datasets.

Original: The results suggest SpectraNet’s performance is not significantly affected by the number of layers in these datasets for several occlusion probabilities.
Revised: The results suggest SpectraNet’s performance is not significantly affected by the number of layers in these datasets for several occlusion probabilities.

Original: On average, SpectraNet 2 ,achieves the best performance.
Revised: On average, SpectraNet 2 ,achieves the best performance.

Original: [Table]
Revised: [Table]