Paper: ProFITi: Probabilistic Forecasting of Irregular Time Series via Conditional Flows (ICLR-2024)

Reviewer comment (Experiments):
In my opinion could be interesting to assess the behavior of the proposed model in several conditions in terms of percentage of missing values, sparsity of the time steps. In fact the irregular time series are very important and to give researcher an evaluation of these aspects could help to understand the quality/limitations of the proposed work.

Author response:
- We agree. We added an ablation study with varying sparsity levels
  (time sparsity and missing values) in Appendix H.5, H.6. Our findings
  show that even in an extremely sparse scenario, where 90% of samples
  (or timepoints) are missing, ProFITi consistently outperforms both
  Neural Flows and GraFITi+.

Reviewer comment (Evaluation):
The authors use likelihood to assess the quality of the forecast also for marginal and point forecast. In that case I think could be useful to have other metrics as RMSE, MAPE or other that is more related to the prediction error.

Author response:
- We **do use MSE** to evaluate models w.r.t. point forecasts (tab. 4).

Reviewer comment (Experiments):
The computational effort of proposed solution is not provided and is not compared to other possible solutions. Some results are provided in appendix but only for a particular dataset.

Author response:
- In the updated version, we included the **runtime per epoch 
   for all models and datasets** in Table 3.
- Our main claim is to provide a **more accurate model**, not to provide a faster
  model (contribution 5). Thus we gave more room to evaluation w.r.t. accuracy
  but w.r.t. scalability.

Reviewer comment (Writing):
§3 - IMTS Query section, in the definition of QA(C), what is the meaning of $|x^{qu}|=|y|$? After, when you talk about NJNL, looks like $|.|$ means length of, but authors should describe clearly the symbols they use.

Author response:
- yes, |.| denotes the length of a sequence. We added a brief section on notation (appendix A).

Reviewer comment (Writing):
In the IMTS probabilistic forecasting problem "(with $min_k t_{n,k}^{qu} > max_i t_{n,i}^{obs}$)" --> the reader is lead to think that $t_{n,k}^{qu}$ is the k-th observation of the n-th query and the same for $t_{n,i}^{obs}$ but a clarification would be better.

Author response:
- You are right. We added a respective note to section 3.

Reviewer comment (Writing):
What is the meaning of $\pi^{-1}$ in equation 4?

Author response:
- for permutations $\pi$, one often writes $x^{\pi}$ to denote the application of the permutation $\pi$ to vector x. 
- $\pi^{-1}$ is permutations that are inverse w.r.t $\pi$. $(x^\pi)^{\pi^{-1}} = x$
- we added a brief section on notation (appendix A).

Reviewer comment (Writing):
In eq. 7 $X_{:,1:|X|-1}$ is the matrix X except the last column?  
In eq. 7 $X_{:,|X|}$ is the last column of matrix X?

Author response:
- yes. We added this, too, to the notation section (appendix A).

Reviewer comment (Experiments):
In the protocol authors said that they use the first 36 hours of observations and forecast next 3 time steps. These time step is, looking at Appendix A, 1 hour for Physionet, 30 minute for MIMIC-III, 1 minute for MIMIC-IV. Is this right? Moreover, is there a way to indicate the time sparsity of considered timeseries in order to understand how the timeseries considered are not uniformly spaced?

Author response:
- Yes, the observation intervals are data set specific.
- We added an additional column "time sparsity" to table 5 that quantifies
  the sparsity of the time series (as proxy measure for its irregularity).

Reviewer comment (Presentation):
The section 6 and figure 3 should be improved in order to make clearer the architecture. In equation 11 the $x_k$ is the $h$ of figure 3? What is the meaning of S in fig. 3?

Author response:
- Yes, $h_k$ are the encoded $x_k$ and replace $x_k$ everywhere (as stated 
  below their definition in paragraph "Query embedding": "take the roles
  of the $x_k$").
- $S$ is the sorting criterion introduced at the end of section 5.
- We added references for all symbols in fig. 3 to their respective definitions to make navigation between formulas and the diagram faster.

Reviewer comment (Presentation):
In Fig. 9 the GraFITi+ returns only a region and not the trajectories? Is it possible to compare the confidence regions of ProFITi and GraFITi+?

Author response:
- Since ProFITi provides non-parametric distribution, it is difficult to plot the confidence regions for multiple points simultaneuously. Hence, although it is interesting, we do not want to pursue the idea.

Reviewer comment (Experiments):
Some general comments (also present in the Weakness section) — In my opinion could be interesting to assess the behavior of the proposed model in several conditions in terms of percentage of missing values, sparsity of the time steps. In fact the irregular time series are very important and to give researcher an evaluation of these aspects could help to understand the quality/limitations of the proposed work.

Author response:
- We agree. We added an ablation study with varying sparsity levels
  (time sparsity and missing values) in Appendix H.5, H.6. Our findings
  show that even in an extremely sparse scenario, where 90% of samples
  (or timepoints) are missing, ProFITi consistently outperforms both
  Neural Flows and GraFITi+.

Reviewer comment (Experiments):
Some general comments (also present in the Weakness section) — The computational effort of proposed solution is not provided and is not compared to other possible solutions. Some results are provided in appendix but only for a particular dataset.

Author response:
- In the updated version, we included the **runtime per epoch 
   for all models and datasets** in Table 3.
- Our main claim is to provide a **more accurate model**, not to provide a faster
  model (contribution 5). Thus we gave more room to evaluation w.r.t. accuracy
  but w.r.t. scalability.

Reviewer comment (Theory):
The requirement of permutation invariance, as proposed by the authors, is questionable in time series analysis and may not be suitable for this domain. The authors argue that a density model should produce equivalent density values when the outputs are swapped, which is a reasonable expectation in the context of static generative models. However, in the time series setting, where the joint distribution of variables $y_1,...,y_K$ occurring at different time steps is considered, the presence of serial dependencies becomes crucial. Unfortunately, the permutation invariant requirement, which treats the order of data points as interchangeable, risks disrupting these vital temporal dependencies. In contrast, if the objective is to forecast the joint distribution of variables occurring at the same time step, the permutation invariant requirement might be deemed necessary.

Author response:
## w1. The requirement of permutation invariance, as proposed by the authors, is questionable in time series analysis

- Permutation invariance is crucial **w.r.t. channels**: it should not matter,
  if you query for channel 1 or channel 2 first. 
- Permutation invariance **w.r.t. time points** is not a necessary property
  of a good model due to the causal aspects you mention. But it is a
  **very successful property** of most state-of-the-art models in vision and
  time series. This motivated us to search for a novel invertible variant
  that can be used in conjunction with normalizing flows.
- All queries carry the **time point as an attribute** ($q_k$ in the paper), 
  like positional encodings in transformers, so that the model can
  detect and represent temporal dependencies, of course.
- We added a respective paragraph to our problem analysis section 3.

Permutation invariance of the queries is not the same as the
conditional decomposition. Permutation variant/dependent
models have an inductive bias to learn correlations between
the 1st, 2nd, 3rd element of a query. Permutation invariant
models do not have this bias, they have to learn the correlation
between queries based on the time point / the positional
encoding alone.

For decades researches have planted a strong inductive bias
based on neighborhoods in their models (e.g., with convolutions),
and we agree that this sounds plausible. However, since the
application of transformers for vision (ViT, Dosovitskiy et al. 2021)
attention mechanisms also have been used to learn those
relations between positions, and these models have been highly
successful, both providing state-of-the-art results and being
highly cited by other researchers (ViT is cited 24823 times in
3 years). 

To the best of our knowledge, there is no way to prove that one approach is better than the other. But we provide extensive empirical evidence that enforcing permutation invariance also for joint probabilistic forecasting of irregular time series with missing values provides the best currently known predictions. We argue by analogy that this is not implausible because large parts of the ML community use similar models in similar contexts (e.g., vision transformers).

References:
- Dosovitskiy, Alexey; Beyer, Lucas; Kolesnikov, Alexander; Weissenborn,
  Dirk; Zhai, Xiaohua; Unterthiner, Thomas; Dehghani, Mostafa;
  Minderer, Matthias; Heigold, Georg; Gelly, Sylvain; Uszkoreit, Jakob:
  "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale".
  ICLR 2021.

Reviewer comment (Writing):
What is the meaning of the L.H.S. in Eqn. (4)? There is a superscript $\pi^{-1}$ on the L.H.S. in Eqn. (4), what does it mean?

Author response:
## q2. What is the meaning of the L.H.S. in Eqn. (4)? There is a superscript $\pi^{-1}$ on the L.H.S. in Eqn. (4), what does it mean?
- for permutations $\pi$, one often writes $x^{\pi}$ to denote the application of the permutation $\pi$ to vector $x$. 
- $\pi^{-1}$ is also a permutation which is inverse of $\pi$.
- here it means, that $f$ is equivariant: if you permute inputs $z$ and $x$, then the outputs of $f$ are permuted the same way. 
- we added a brief section on notation (Appendix A).

Reviewer comment (Experiments):
The choice of $\epsilon$ in Eqn. (6) should affect the training results. Should it be determined before training? Or should it be termed as a hyperparameter during training? Should there be an investigation about the choice of $\epsilon$?

Author response:
## q3. The choice of $\epsilon$ in Eqn. (6) should affect the training results. Should it be determined before training? Or should it be termed as a hyperparameter during training? Should there be an investigation about the choice of?
- yes, you are right, it does. It is a hyperparameter.
- we clarified this next to Eq. 6. 
- we use $A^\text{tri}$ (Eq. 9 of revised version) as a vector field in all our experiments and set $\epsilon = 0.1$. It is also shown in Figure 2. We corrected the typo in Eq. 15 in the revised version (from ISA to SITA) which could have caused the confusion.
- we added an ablation study for varying $\epsilon$ in $A^{\text{tri}}$ (appendix H.4).

Reviewer comment (Experiments):
Why use the GraFITi model to encode the historical data? Compared to the prevalent model used to model irregular time series, such as GRU-ODE-Bayes [2] or Neural CDE [3], what is the advantage of GraFITi?

Author response:
## q4. Why use the GraFITi model to encode the historical data? Compared to the prevalent model used to model irregular time series, such as GRU-ODE-Bayes [2] or Neural CDE [3], what is the advantage of GraFITi?
- GRU-ODE-Bayes, CDE and other models are forecasting models, not encoders. They yield for each query a scalar, the predicted value, not an embedding vector. While in principle it would be possible to use them as (scalar) encoders, due to their limitations to a single dimension we did not follow up on this idea.
- We opted for GraFITi, because 
  1. it has been shown to outperform many other models, incl. GRU-ODE-Bayes, NeuralFlow, CRU etc. (see their table 2) and
  2. it is very fast.
- We added this justification to section 6 / subsection "query embedding".

Reviewer comment (Evaluation):
Most literature use CRPS and CRPS_sum to evaluate the performance, e.g., [1, 4-6]. Why authors do not follow the literature?

Author response:
## q5. Most literature use CRPS and CRPS_sum to evaluate the performance, e.g., [1, 4-6]. Why authors do not follow the literature?
- Most literature on forecasting **irregularly sampled time series**
  uses loglikelihood for comparison (De Brouwer et. al., 2019; Bilos et. al., 2021),
  we do the same to be directly comparable.
- You are right, that the literature on **predicting marginal distributions for 
  regularly sampled time series** uses CRPS and/or CRPS_sum. But both measures
  do not cover joint distributions. So we did not use them.
- We add a delineation to the evaluation section 7.

Reviewer comment (Evaluation):
Evaluation covers only three datasets from very similar domain.

Author response:
- We agree. We added experiments on USHCN, an IMTS climate dataset 
  that also has been previously used in the literature (De Brouwer et al. 2019, 
  Schirmer et al. 2022, Yalavarthi et al. 2023) to section 7.

Reviewer comment (Presentation):
Please adjust the abstract in line with ICLR formatting instructions.

Author response:
- Fixed. Thanks!

Reviewer comment (Theory):
It would be beneficial to study what kind of distributions can be modelled with the proposed flows.

Author response:
- ProFITi is a normalizing flow model, as such it is not limited to a distribution
  with a fixed shape like the Gaussians. 
- However, quantifying the exact expressive power of normalizing flow 
  models is an active research topic of its own and beyond the scope of our
  paper (see, e.g., Kong/Chaudhuri 2020):
- We added a note to section 2 Literature review.