Paper: Asynchronous Graph Generators (ICLR-2024)

Reviewer comment (Novelty):
Although the problem itself is interesting, the contribution of the paper seems to be incremental. For instance, if I understand right, the model can only predict/impute measurement (i.e., $y$). However, an essential question in asynchronous event modeling is how to effectively predict the time/attribute of the next or future events.

Author response:
**Contribution is incremental:** 

We respectfully disagree with this statement. Our work presents a thorough revision of the SOTA on graph NNs for time series and position our contribution very clearly. We summarise our contributions again here: 

 - The interpretation of time series as a directed graph which is free from assumptions of temporal regularity that typically constrain prior architectures such as RNNs---see **BRITS**: _Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. NeurIPS 31. Curran Associates, Inc., 2018._)
 - The use of this geometric interpretation (inspired by _Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges._) to derive a novel class of architectures that leverages this asynchronous graph through learnt graph attention (see _Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637, 2020._)
- The application of this graph to multichannel time series, in a computationally efficient manner, by casting the problem from a heterogenous graph to a homogeneous graph via novel channel embedding (referred to as metadata embedding in the paper in Sec. 3)
- The extension of SOTA (see _Petar Veličković. Everything is connected: Graph neural networks. Current opinion in structural biology, 79:102538, April 2023. ISSN 0959-440X. doi: 10.1016/j.sbi.2023.102538._) equating transformers to GNNs and applying this to the asynchronous representation via a novel causal temporal masking of dot product self attention
- A novel methodology of transductive node generation via cross attention using a learnable representation of metadata and temporal embeddings. Time and channel can be chosen arbitrarily to conditionally _generate any_ node in the asynchronous graph allowing a flexible methodology for imputation and prediction (simply choose a time that succeeds the final measurement to cater for prediction)  

**The model can only impute measurement but not predict time/attribute future events.** Perhaps the aim of our work was not clear to the Reviewer: As stated in the first lines of the paper, our work focuses on data imputation and not on predicting when the next measurement will occur. This setup is not unique to our work but shared by all methods considered in the paper and the SOTA against which we compare:

- SSGAN: _Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative
semi-supervised learning for multivariate time series imputation. In AAAI, volume 35, pp. 8983–8991, 2021._ 
- BRITS: _Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In NeurIPS 31. Curran Associates, Inc., 2018._

Reviewer comment (Novelty):
The model architecture is not novel. Most of the paper discusses embedding and attention structure, which have been well-studied in previous works. I don't think the contribution

Author response:
**Architecture is not novel, embedding and attention have been well-studied in previous works.** We would to respectfully invite the Reviewer to be more open-minded about what a contribution to ML research is. Currently, most neural architectures are based on the attention mechanism, which has enabled clear advantages in many applications. Saying that using attention is not novel is equivalent to saying that any architecture using convolutional layers, skip connections or recurrences are invalid since those components are known. Furthermore, our work is not aimed at *studying attention and embeddings*, but to design and extend specific instantiations of these concepts that are useful to our setup. For instance: i) the temporal (Fourier-based) embedding aims to extract spectral information, ii) the metadata embedding allows flexible (homogeneous) representations of heterogeneous graphs (those with multiple sources), and iii) the attention heads aim to learn the graph dependencies among the nodes values, timestamps and metadata  (see Sec 3).

Reviewer comment (Experiments):
More numerical experiments should be expected to highlight the effectiveness/characteristics/benefits of the model, e.g., simulation or ablation studies. The current results seem to be a black box.

Author response:
**More experiments expected such as simulation or ablation studies. The current results seem to be a black box.** We acknowledge that the more experiments the better, but we do not understand what model features need to be clarified via more experiments in the view of the Reviewer, as they were not specified. We have considered exactly the same setup of all the SOTA benchmarks considered (listed above) and replicated their experiments, while also performing an ablation study for data augmentation in Sec 6. Lastly, we are unsure how to take the comment of our model being a _black box_: we have detailed as much as possible the architecture and training procedure. Perhaps our model is a _black box_ as much as any other neural network model.

Reviewer comment (Presentation):
Only data imputation is illustrated in Fig. 2, and based on that, it is a bit hard to imagine the situation of prediction (i.e., new nodes come after known temporal embedding). For example, if $L=4$ in Fig. 2, which batch should nodes 5 and 6 go to?

Author response:
**How to perform prediction.** The AGG performs prediction/imputation as transductive node generation; this is achieved by conditioning the cross-attention block on an arbitrary selection (user-defined) of the metadata and time. The selections chosen by the user are encoded via the learned embedding representation $(t2v(\tau), embed(m_t))$ (eq 8) and then used in the cross attention (Sec 3.4) AGG block along with the embeddings of the input graph to compute (predict) the value of the new node in the asynchronous graph. Furthermore, as stated in 3rd paragraph of Sec 3 in the paper: _"...If the encoding places the new node within the temporal “neighbourhood” of the other nodes in the graph, we refer to data imputation, whereas if the new node comes after the known temporal encodings we refer to prediction"_. Explicitly, if a time value is chosen that _succeeds_ the inputs, we perform prediction. Following the question of the Reviewer, we have included a prediction example on the Beijing dataset in Appendix A.4, emphasising that this required no modification of the existing architecture. 

Regarding the Reviewers question about context length in Figure 2, extending the context length to $L=4$ would use nodes $0,1,3,4$ in the input. Also, generating node 2 would be considered _imputation_, while generating nodes 5 and 6 would be considered _prediction_. Alternatively, nodes 7,8,10,11 could be used to generate both nodes 5 and 6, in which case we would refer to _imputation_. Note that the flexible representation of the conditional generation allows this type of arbitrary generation of nodes (which represent new samples). Note the context length of $L=3$ is simply demonstrative, the context length is a hyperparameter and is typically chosen based on the computational resources available ($L=100$ was used in our experiments).

Reviewer comment (Experiments):
Is there any real example in the experiments about prediction?

Author response:
**Real world examples.** All our data come from the real datasets Beijing Air Quality, PhysioNet Challenge 2012, and UCI localisation. The reason we considered those datasets is because all the SOTA benchmarks considered are tested on them and thus we can directly report objective quantitative comparisons of AGG. See:


- SSGAN: _Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative
semi-supervised learning for multivariate time series imputation. In AAAI, volume 35, pp. 8983–8991, 2021._ 
- BRITS: _Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In NeurIPS 31. Curran Associates, Inc., 2018._

Reviewer comment (Experiments):
As the key advantage of this work (when compared to the current baselines) is the use of attention, I think it is problematic that the authors have not compared extensively to attention-based temporal data imputation techniques. Particularly, [1] seems to be a very similar model and work, as they also evaluate on the same datasets. See their Table 1; AGG outperforms theirs in 10% and 50% missing data for PhysioNet, but they claim superior performance for 90%. The Beijing Air Quality data seem to be scaled differently, but you should also compare to their result. In summary, a comparison should be made empirically and the differences in the methodology should be highlighted to other attention-based works for data imputation in temporal data.

Author response:
1. We thank the reviewer for connecting our work with [1], as we were not aware of this work. We will endeavor added a comparison with [1] in the revised manuscript. Your suggestion of including a more focused comparison with attention-based methods is also well taken.

Reviewer comment (Experiments):
It seems a strong assumption that the time and (especially) metadata of unknown targets are known - can you demonstrate the ability to predict the metadata, at least? Regardless, I think it is a bit of a limitation that the timestamp of the imputed observation must be provided.

Author response:
From the reviewers comments we can see that we failed to explain how exactly the metadata is obtained and why it is valid assumption that this is known for the prediction step, this is also related to why we refer to the graph as heterogeneous and by including the metadata as an embedding we can cast the problem to a heterogeneous one. We believe a comprehensive example will help to clarify this point. The Beijing data set, for example consists of 12 weather stations seperated geographically:
$$ station \in \\{ Aotizhongxin,Changping,Dingling, Dongsi, Guanyuan,Gucheng, \ldots, Wanshouxigong \\} $$

and for each of these stations they provided different sensor measurements (from different types of sensors often simultaneously, although often their values are missing):
$$ sensor \in \\{
    PM2.5,
    PM10,
    SO2,
    NO2,
    CO,
    O3,
    TEMP,
    PRES,
    DEWP,
    RAIN,
    WSPM
\\} $$
 There is also additional categorical metadata for each station, this is the $WindDirection$ which is the cardinal directions 
 $$ WindDirection \in \\{NNW, E, NW, WNW, N, ENE,
    NNE,
    W,
    NE,
    SSW,
    ESE,
    SE,
    S,
    SSE,
    SW,
    WSW,
    None
\\}
$$ The features $y$ are the measurements of any $sensor$ at any location $station$. One way to express the interconnection between physical locations and sensors is to use a heterogeneous graph, where the types of nodes are the different $station$ and $sensor$ combinations and the edges are the measurements $y$. 

The $WindDirection$ is a categorical feature of each node and is used to extend the information of the features through a categorical embedding. The metadata is known for all stations and sensors (as it is implicit characteristics of each node), but the measurements $y$ are not known for all sensors at all stations (as they have temporal, spatial, type dependence) and are the values we want to predict, but we need to know which $station$ and $sensor$ in order to do so, which makes sense intuitively as at time $t$ how could the model know for which $sensor$ and at which $station$ we wish to predict its value, logically the value of $PM2.5$ will vary across physical locations that are separated by many kms and many different sensor give values at the same time, thus it makes sense to _condition_ the generation step. 

Finally, the $timestamp$ is an independent value that is simply the time of the measurement relative to the start of the batch, there is no restriction in its value thus we can select this arbitrarily, and it is then encoded through a learnt `time2vec` embedding module (to capture frequency dynamics). Of course, we choose time values for which we have measurements so that we can train the model but generation is not bound by such restrictions.

We hope this example clarifies the assumptions we make and why we refer to the graph as heterogeneous. The problem of modelling a heterogeneous graph is that we would require a different set of weights for each node type (we induces data sparsity problems), which is why we embed the node metadata and instead treat it as a homogeneous graph where the sensors type and the geographical location are encoded in the node embedding and concatenated along with the measurement $y$ (this alows the model to leverage redundancies between different node types under a common set of graphs or "attention heads"). This is also why “the input layer of dimension $5 \times 16$”, the 5 is the number of separate embedding types the feature, station, type, time and categorical (wind direction in this case) data and the 16 is the dimension of the resulting embedding. To ease the notational burden (perhaps to the detriment of clarity) we referred to station, type and categorical data collectively as "metadata".

The generation step simply consists on selecting of which "metadata" we want to condition on in order to predict the $y$ value, and the model will output the generated value, there is no rational for predicting the metadata as it is simply which part of the "heterogeneous graph" we want to generate.

Reviewer comment (Experiments):
A minor comment: perhaps outlining the benefits/properties of each baseline would be helpful.

Author response:
4. We will endeavor to clarify the purpose and details of the baselines in the revised manuscript.

Reviewer comment (Experiments):
Please comment on the difference between your approach and other attention-based approaches for temporal data imputation, and compare to these apparently strong baselines.

Author response:
1. We thank the reviewer for connecting our work with [1], as we were not aware of this work. We will endeavor added a comparison with [1] in the revised manuscript. Your suggestion of including a more focused comparison with attention-based methods is also well taken.

Reviewer comment (Experiments):
Consider continuous-time models, such as a small variation of the GP-VAE baseline and perhaps a point process type of imputation model.

Author response:
2. We thank the reviewer for the suggestion of comparing with continuous-time imputation techniques and agree that this is a rather unexplored area of research. We will endeavor to include a comparison with GP-VAE and PILES in the revised manuscript. We also agree that the ability to impute continuous-time data is a key advantage of our method and will endeavor to highlight this in the revised manuscript, we appreciate the insight of the reviewer on this point.

Reviewer comment (Experiments):
Outline the purpose and details of the baselines.

Author response:
4. We will endeavor to clarify the purpose and details of the baselines in the revised manuscript.

Reviewer comment (Theory):
Line above Eq (3) “AGG is a heterogeneous graph” but it is actually homogeneous?

Author response:
Please refer to our response detailing the heterogeneous graph assumptions and why they are valid, we will endeavor to clarify this in the revised manuscript but we maintain that this is in fact a heterogeneous graph problem to which we cast to a homogeneous graph problem by embedding the metadata.

Reviewer comment (Presentation):
After cross multi-head attention in Figure 3, a residual connection is shown between the output of CrossMultiHead and \(h_l\). However, Eq (10) shows the residual connection between the output of CrossMultiHead and \(g_0\).

Author response:
7. Thank you for pointing this out, yes the figure is incorrect and the residual connection should be between the output of CrossMultiHead and $g_0$ as shown in Eq (10). We will endeavor to correct this in the revised manuscript.

Reviewer comment (Reproducibility):
In implementation details “the input layer of dimension 5 \times 16” where does the 5 come from in this case?

Author response:
Please refer to our detailed heterogeneous graph example for a detailed clarification. Briefly the input layer of dimension $5 \times 16$ is the number of separate embedding types the feature, station, type, time and categorical (wind direction in this case) data and the 16 is the dimension of the resulting embedding.

Reviewer comment (Writing):
The reviewer believes a good research paper should be educative to general audience; they did look at Cao's work on RNN for time series imputation and find their problem set-up and proposed approach easy to understand. Unfortunately, the current form of this paper makes it really difficult for readers without certain background knowledge to understand the setting and the contribution.

Author response:
We will endeavour to make the next iteration of this manuscript more accessible to the general audience.

Reviewer comment (Evaluation):
Given that this paper is purely empirical, the numerical experiments are the most important part to verify the performance. Table 1 may benefit from including uncertainty quantification (in the meanwhile, the reviewer acknowledges that the improvement is quite significant).

Author response:
Thank you for the insight we agree uncertainty quantification would improve the imputation section, for the post imputation classification and regression we performed a 5 fold cross validation which indeed quantifies the uncertainty so some degree.

Reviewer comment (Writing):
Terminology should be used more carefully: the term "causal" is used multiple times (and perhaps that is the reason why the reviewer gets invited to review this paper); however, it seems that "causal" merely refers to temporal order, which is "Granger causal" and means correlation from past to the future. The reviewer suggests using simple terms like temporal order directly.

Author response:
We appreciate the connection to "Granger" causality, we had not considered this definition, and the reviewer is correct the "causal" refers to the "temporal order", although the underlying design is to capture (learn through graph attention) the phase dynamics between different measurements i.e. that the measurement from sensor $y_k$ at time $t-2$ is related to sensor $y_l$ at time $t$ by some hidden dynamics (which are related by some phase shift in the signals) of which we wish to learn through graph attention.

Reviewer comment (Evaluation):
Scaling might be one major issue when the dimension is high and the time horizon is long (since there must be a really huge graph to represent the multivariate time series).

Author response:
We will endeavour to explore the limitations of architecture wrt to high dimensionality and longer time horizons, thank you for the suggestion.

Reviewer comment (Theory):
In Fig. 1 (c), the proposed method used future to predict past. The reviewer is fine with using the expressive power of neural networks to learn a latent causal representation, but a structure shown in Fig. 1 (c) seems to imply that the latent data generation mechanism depends on the future which is clearly wrong. Can the authors justify that? (PS: in Granger causal literature, one famous example is that "buying Christmas tree" Granger causes "Christmas". However, this example cannot justify the proposed structure since there should be a latent variable "knowing Christmas will be on 12/25" captured by the latent data generating process).

Author response:
Figure 1b) shows the temporal ordering of the masking of the graph, which respects the ordering of the signals, this input graph is then encoded along with the signals through the AGG which results on a set of encoded signals $h$ which a used to either impute or predict any other node. In the case of figure 1c) this is demonstrating imputation (filling in missing values) which indeed uses the future value to fill in the missing sensor measurements. This is justified as the purpose is to use all available information to infer the missing value. In the prediction case this indeed would be a violation but would not occur as the generated node would be after the encoded inputs in order to be prediction. We appreciate the reviewers question.