Paper: Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations (ICLR-2024)

Reviewer comment (Presentation):
The conclusion/discussion was quite brief. I would have loved to read more about limitations and the approach for extending this to the multivariate case.

Author response:
Thank you for the question. We will provide more insight on why our model would struggle with heterogeneous multivariate time series and what can be done to overcome this challenge. 

- As mentioned in the conclusion, our empirical results have shown that TimeFlow is effective at handling homogeneous multivariate time series but would struggle at modeling time series with heterogeneous distributions. The key idea is that the INR learns shared parameters representing the general statistics of a family of time series and the modulation derived from the learned code $z$ adapts the INR to each sample specificity. Intuitively, shared statistics correspond to "low" frequencies, while modulations learn time series-specific "high" frequencies. If the "low" frequencies completely differ between time series (for example, let's consider a dataset where one sensor measures humidity, another sensor measures temperature, and the third sensor measures visibility), the shift modulation mechanism alone would not be sufficient to complement the statistics learned via the shared parameters.
- One solution to overcome this challenge is to have one INR per time series distribution. All INRs can be modulated with the same vector $z$ that characterizes shared information, e.g., the location. This method is flexible but expensive if there are many different time series distributions.

Reviewer comment (Presentation):
Section 3.4 was a strong inclusion of rationale for their authors' choices, but the disjoint list of conclusions and redirection to the Appendix was weak. Perhaps some (unnecessary) details of the datasets could be left to the appendix to provide space for more description of the actual method.

Author response:
We agree that the discussion on architecture details could be more structured, and removing some details of the dataset would leave us the necessary space. We have updated the article accordingly.

Reviewer comment (Theory):
The 'efficient adaptation in latent space' is interesting. Is there anything to be learned about the structure of the latent spaces and how they are modified between potentially similar datasets? Which could explain perhaps why other models require a full retraining.

Author response:
Thanks for this suggestion. In order to explore and to understand the structure of the latent space, we performed a series of three new experiments with results detailed in Appendix E:

1. We explore how the latent space behaves between two learned codes, $z_1$ and $z_2$, which are representations of two time series $x_1$ and $x_2$. We propose in Figure 13 to visualize how new latent representations $z_{\lambda} = \lambda z_1 + (1 - \lambda) z_2$  are decoded in the time-series domain ($f_{\theta, h_w({ z_{\lambda})}}(t)$ values). We observe that the interpolation path between two codes yields a smooth transition in the time series domain. This suggests that the latent space is smooth and well-structured, and that the learned representations captured meaningful features of the time series, which could explain TimeFlow's generalization property.

2. We tried to interpret each dimension of the latent space independently. To do so, we perturbed a given modulation dimension (by adding Gaussian noise) only for a given layer of the INR. We then observe the difference in the time domain between the unperturbed and the perturbed time series. In Figures 14, 15, and 16, we observe three different effects on the time domain for three different perturbations of the shift modulations. Drawing conclusions for a specific code dimension for one particular INR layer is challenging.

3. In Appendix E.3, we visualize two time series distributions in latent space. The two distributions differ by a time shift. According to Figure 17, the larger the time shift the more the distributions are separable in the latent space. However, we note that this separability does not affect the generalization performance of TimeFlow over time, as shown by the results presented in Table 2 and Table 18.

Reviewer comment (Evaluation):
Would this work for imputation or forecasting not on a grid? As in, could forecasted data be easily predicted on an arbitrary grid? Is this related to the discrete performance in Table 2?

Author response:
TimeFlow operates without needing a specific grid; it simply requires access to timestamps and corresponding values. In the experiments we used grids only for allowing comparisons between methods and also because most datasets are provided on regular grids.

Reviewer comment (Novelty):
Although the domain of the application (i.e., time-series modeling) is new and the proposed design brings an idea of the latent state evolution (in forecasting), the novelty seems to be limited. The overall architectural design follows the FFN architecture (Tancik, et al, 2020) without any consideration on how to handle multivariate time-series. Also, the idea of the latent modulation and the meta-learning-based training algorithm largely follow the previous approach (Dupont, et al, 2022). Finally, a similar idea of employing temporally-refined latent variables has been explored in (Yin, et al, 2023).

Author response:
Thanks, we will try to highlight the differences with previous work. As indicated in the general comment, we think that our contributions involve non trivial developments and differ significantly from existing works as detailed below.

1. Comparison with [1]. In [1], the focus is on self-supervised learning: INR is only used for learning representations from static images/3D shapes that are later used for downstream tasks (classification and generation). For TimeFlow, INR is not used for learning representations but is designed for modeling contexts and dynamics in a continuous manner: this is a significantly different framework.

 Concerning the specificities of  the meta-learning implementations in [1] and TimeFlow, please refer to general comment entitled "Unique Adaptation of Meta-Learning Principles".


2. Comparison with [2]. The frequencies embedding presented in [2] is only one of the possible INR implementations. TimeFlow uses the NERF encoding [3], which brings simplicity and good performance (we have an ablation on this topic in the paper; please see Table 4). Overall, the key is to capture multiple frequencies, and there are various alternative implementations that can achieve this. We chose one of them.

3. Comparison with [4]. In [4], authors make use of spatial INRs and the dynamics is encoded via a NeuralODE solver. Besides, the two components are trained sequentially and not end to end as we do so that the codes are fixed and not refined during training. This is essentially different from TimeFlow were for each new dynamics, the latent code is updated according to the look-back window to capture the local dynamics.

[1] E. Dupont, et al. From data to functa: Your data point is a function and you can treat it like one. ICML 2022.

[2] Tancik, Matthew, et al. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS 2020.

[3] Mildenhall, Ben, et al. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM.

[4] Y. Yin, et al. Continuous pde dynamics forecasting with implicit neural representations. ICLR 2023.

Reviewer comment (Experiments):
For imputation, it is natural to assume that there is a given set of measurements for a new time-series, and to set up the goal to fill-in unseen data via imputation. For forecasting, however, the assumption of having a separate training period and a look-back window raises some concerns. Having a separate look-back window suggests that the method needs to wait until the new observations are collected to make forecasting. Some of the datasets that are considered in the paper have hourly sampling rate and this time gap might provide enough time to fine-tune other baseline models (with many model parameters, e.g., Transformers). If the ultimate goal is to achieve accurate prediction, with the given time period (an hour), fine-tuning those baselines with a new observation may provide better prediction results.

Author response:
If we understand correctly this question, concerns arise from Tables 2 and 18, where we train over a given period and then test over multiple periods in the future. Indeed we make the assumption that the models will not be retrained during a given operational period, which corresponds we believe to most real world applications. This is a classical forecasting framework. Then in any case, forecasting is performed at a given horizon, from an adjacent look-back window. This holds  for any forecasting model.
Please tell us if we misunderstood your remark.

Reviewer comment (Experiments):
Another concern is fairness on the comparisons. Although it is just 3 gradient steps, auto-decoding is considered as solving an optimization problem to fine-tune the model for the new observations. What happens if the small portion of the other baselines (e.g., the last layer) is fine-tuned during the inference? For example, in forecasting, the model can be fine-tuned after making predictions on the current sliding window and then make predictions on the next sliding window with the updated models.

Author response:
Auto-decoding does not perform any fine tuning and all the models including baselines are evaluated in the same conditions. In a forecasting setting, the objective is to forecast future values (horizon) conditioned on past ones (look-back window). A forecaster could make use of any encoding of the look-back information. For example one could use the look-back window values as such with no change, or one can encode the look-back values in a compact representation used as input for a forecaster (a simple compact representation would be for example a sum of the look-back values). Auto-decoding is just another way of computing this encoding. In order to better detail why auto-encoding does not perform any form of fine tuning, let us now try an alternative rewriting of the auto-decoding that might make things clearer. Let $t^{L}$ stands for the look-back window timestamps and $t^{H}$ stands for the horizon timestamps. We consider for sake of simplicity only one gradient step to encode $z$ according the look-back window values $x^{L}$. The TimeFlow forecaster can be written as a deterministic function of the look-back window:
$$
    f(t^{H}; \theta, w, z(x^{L})) \; \text{where} \; z(x^{L}) = z^{(0)} - \alpha \nabla_{z}\Vert f_{\theta, h_{w}(z^{(0)})}(t^{L}) - x^{L}\Vert^{2}_{2} 
$$
Making 1, 3, or 10 adaptation gradient steps uniquely changes the complexity of the function encoding the code $z$.

As a side note, DeepTime also uses this type of encoding. Time-step-based models that don't have a structure with an explicit set encoder need to pass the look-back window information somehow.

Reviewer comment (Evaluation):
Although the method seems to provide accurate predictions both in imputation and forecasting, the method seems to struggle in predicting peaks accurately. In many applications, predicting peaks accurately would have more importance than simply minimizing MSEs (e.g., to properly prepare the electricity supply or properly set up the cost during the peak time period). Based on the eye-ball examination (Figure 5 for example), the model does not seem to provide accurate predictions in peak values.

Author response:
You are right; in many real-world applications, accurate peak prediction would be more important than minimizing MSEs.

We believe that this kind of "smooth" prediction is more a matter of the choice of the optimized loss (in our case, MSE) than the model. For example, in Figure 10, where we compare the quality of the TimeFlow and PatchTST predictions, both methods have trouble catching the peaks.

If the observations are regularly spaced, one way to overcome this challenge is to train the models with more appropriate but more expensive losses, such as Soft-DTW [1] or DILATE [2], which are differentiable. If the observations are not regularly spaced, to the best of our knowledge, finding a differentiable loss that can penalize shape pattern error is an open problem in time series.

[1] M. Cuturi, et al. Soft-dtw: a differentiable loss function for time-series. ICML 2017.

[2] V. Le Guen, et al. Shape and time distortion loss for training deep time series forecasting models. Neurips 2019.

Reviewer comment (Experiments):
As mentioned in the weakness section, could the author provide more justifications for performing auto-decoding while other baselines are used only for inference? Also, could the authors mention more about the fairness of the comparisons?

Author response:
Please see answer to Weakness 3.

Reviewer comment (Experiments):
Appendix section D, Tables 13 and 14 emphasize the performance degradation in Transformers models as the testing window is far apart from the training period. Would there be realistic cases where we have only the old history for training, measurements are stopped for a while, and collecting results regularly again afterwards?

Author response:
Please see answer to Weakness 2.

Reviewer comment (Experiments):
In Appendix, the paper provides experimental results for varying dimensionalities on the latent vectors and the number of gradient steps in auto-decoding during the inference. However, these experiments are performed in a limited experiment setting. Could the authors provide more insight on the effect of these hyper-parameters? For example, Table 6 provides the results with a specific dataset (Electricity) with a horizon length 720 and a look-back window length 512 and essentially says there will be no improvement after 10 or 50 gradient steps. Would this observation be valid for other datasets, for other sizes of windows? Also, what happens if the dimensionality of the latent vector is changed? Are these results also obtained by the multiple number of runs?

Author response:
Thanks for the question, you are right. It is important to perform these ablations across all datasets and horizons, we have then added several new experiments as detailed below:

- Regarding the ablation on the $z$ dimension. In Appendix A.2.2 we added Table 5, which investigates the impact of $z$ dimensionality on the forecasting performance of TimeFlow. We performed experiments on all the datasets for each horizon with different $z$ dimensions (32, 64, 128, 256). The results suggest that a $z$ dimension of 128 is a reasonable compromise but only optimal for some settings. Moreover, even though the choice of $z$ dimension seems important, it doesn't critically impact the MAE error for the forecasting task.
- Regarding the number of gradient steps within the inner loop. In Appendix A.2.3 we have added Tables 7, 8, and 9. In Table 7, TimeFlow is trained with a single gradient step in the inner loop and tested with various gradient steps (1, 3, 10, 50) in the inner loop at inference. We evaluate this over all datasets and horizons. Tables 8 and 9 show the same results, where the number of gradient steps in the inner loop during training is 3 and 10, respectively. 
Two conclusions can be drawn from these results: 
    - Keeping the number of gradient steps the same for training and inference is a good option.
    - Three gradient steps in the inner loop seems to be a good choice regarding MAE results while being faster than ten gradient steps.

Reviewer comment (Experiments):
Could the authors also provide justifications on not to include other time-continuous models for their baselines? Such as neural ODEs and their variants for irregular time-series modeling?

Author response:
Neural ODEs and their variants for irregular time-series modeling have been shown to be outperformed by CSDI [1] and mTAN [2] for the imputation task. CSDI and mTAN are, respectively, discrete and continuous methods and been re-implemented in our work and are used as baselines. In addition, we would like to highlight that we have re-implemented 11 recent deep learning baselines in this paper.

[1]: Tashiro, Yusuke, et al. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. NeurIPS 2021.

[2]: Shukla, Satya Narayan, and Benjamin M. Marlin. "Multi-time attention networks for irregularly sampled time series." ICLR 2021.

Reviewer comment (Novelty):
The approach has limited novelty since it's mostly building upon known previous work. This same architecture can be applied to images, point clouds, and so on. Although the discussion of implementation choices is a nice addition, they are again not necessarily time series dependent.

Author response:
As already detailed we respectfully disagree with this argument. As detailed in the general comment and in response to strength 0, while the individual blocks of TimeFlow may be already known, their use and combination for time series involve original developments. Notably, the INR algorithms designed for images, point clouds, and similar domains are not directly applicable to TimeFlow due to their lack of consideration for temporal dynamics.

Reviewer comment (Evaluation):
According to Table 16, the model is more costly compared to already costly transformer-based models.

Author response:
Table 20 (formerly 16) shows the inference results. Indeed, due to code adaptation, TimeFlow takes longer at inference than transformers. However, as specified in section A.2.3, TimeFlow can infer $321$ (samples) $* (720 + 512)$ timestamps, which represents around 400k values in less than 0.2 seconds on a single GPU (NVIDIA TITAN RTX GPU 24Go). This performance looks acceptable for real-world applications.

On the memory consumption side, as shown in Table 19, TimeFlow has half the weight of Informer and Autoformer and 2 to 10 times less than PatchTST.

Reviewer comment (Novelty):
The biggest drawbacks of this paper are lack of novelty, not so stellar results and not applying this method to actual continuous-in-time data.

Author response:
Please see previous answers.

Reviewer comment (Evaluation):
Can you explain why the results in Table 13 and 14 differ for AutoFromer and Informer?

Author response:
Table 17 (formerly Table 13) showcases results for the setting where the test window is adjacent to the training period. For Figure 7, this means that the test window is only the test period n°1. Table 18 (formerly Table 14) showcases results where the test windows are disjoint from the training period. For Figure 7, this means that the test windows are test periods n°2, n°3, n°4 and n°5. The results for each model may differ between these two tables because the test periods are different. However, we can see that while the other methods experience a slight increase in their MAE score, Informer and AutoFormer experience a significant deterioration in their forecasting performance. These methods seem to generalize poorly when tested on new time periods.