Paper: Beyond Spatio-Temporal Representations: Evolving Fourier Transform for Temporal Graphs (ICLR-2024)

Reviewer comment (Theory):
The generalization of the proposed method needs to be clarified further. Even considering Theorem 1, the proposed method depends on the approximate \Psi_{AD}, the eigendecomposition of L_{J_d}. It is unclear whether any limitations associated with this will be introduced.

Author response:
-  One known limitation of eigendecomposition is that it is invariant to the sign/direction of the basis vector. This may cause issues with learning since the basis is not normalized. This can be resolved by the sign and basis invariant methods [1]. Moreover \Psi_{AD} is not designed to handle signed and directed graphs. Future works can explore how to generalize the laplacian as well as the proposed EFT to generic signed and directed graphs using techniques proposed in [2] and [3]. Do you recommend we add this discussion to the main paper?

Reviewer comment (Writing):
In section 5.1, ”However, the same holds true for higher dimensions as well by conducting the EFT dimension-wise.” What does the dimension-wise mean?

Author response:
- Dimension-wise means the feature dimension of a node. Each node may have a multidimensional signal residing on it and the EFT can be independently applied to each channel or dimension for the dynamic graph. We shall add this in the revised version of the paper.

Reviewer comment (Novelty):
The idea of decoupling spatial and temporal domains in spatio-temporal data analysis is not new, especially in the spatio-temporal graph neural networks (ST-GCN) community. Most existing ST-GCN models, whether from empirical perspectives [1, 2, 3] or theoretical perspectives [4, 5], share the idea of graph-time decoupling to reduce computational complexity. However, only a handful of prior works on ST-GCNs have been mentioned and discussed in the paper. Even though the authors focus on graphs that change with time, which most ST-GCNs are not directly applicable to since they deal with static graphs, related discussions should be added so that the readers are clearer about the contribution of this paper.

Author response:
> P1: Regarding related works
- We completely agree here and also share the same sentiments. We have incorporated and discussed the related works in the revised version of the paper.

Reviewer comment (Presentation):
For the approximation of EFT to work, two extra assumptions are needed: 1) The rate of change of the graph with time is bounded; 2) The eigenvalues of the graph Laplacian at any given timestep and between timesteps have a multiplicity of 1. These assumptions are important (and intuitive) for the EFT to work and should be stated in the main text instead of the appendix to help readers better understand the limits of the approach.

Author response:
> P2:  I think it is better to state these assumptions in the main text 
- Thank you for your feedback on the presentation. We have incorporated the changes in the revised version of the paper.

Reviewer comment (Writing):
The paper is dense and difficult to read at times. It is a novel work but could have been presented in a more elaborated way by providing more examples, intuitions, and connections to FT and GFT via examples.

Author response:
- Thank you for your feedback on the presentation. We have incorporated the changes in the revised version of the paper, in section 5 page 6. Regarding the example for showing connection to dft and gft we think the example of ring of ring graphs perfectly is able to capture the phenomena. Do you suggest we incorporate it in the paper? Upon your confirmation, we will add it to the final camera-ready version of the paper as a running example.

Reviewer comment (Writing):
A table or example for clarifying the notations will significantly improve the readability. Please provide explanations/intuitions of all the terms in Equation (4) when the EFT is defined first.

Author response:
- We appreciate your suggestion to include a table for clarity. Unfortunately, due to page constraints, we could not add a table. However, we address it by providing detailed notations in Section 3. If further clarification is required, we are open to suggestions.

Reviewer comment (Presentation):
The coupling of filtering in vertex and time domains is explained in Appendix C.2, which should be moved to the main text.

Author response:
- We have incorporated the changes to the main paper (cf. sec 5.1).

Reviewer comment (Presentation):
Graph signal variation equations can be displayed with numbers since it is the basic term used in further analysis.

Author response:
-  We have added this recommended modification (equation 3, page 4).

Reviewer comment (Writing):
It would help the readers if more insights are given in terms of small toy examples.

Author response:
- For a ring graph we can show that the laplacian is a circulant matrix and the eigendecomposition gives the basis vectors of the DFT(classical Fourier transform) matrix with the eigenvalues the corresponding frequencies. For a ring of ring graphs, we can show (using property 1) that the EFT matrix comes out to be the Kronecker product of two DFT matrices (one along the time domain and one along the vertex domain). Similarly, the harmonics will be the Kronecker sum of the harmonics along the individual domains. To remark, we can say that the GFT is a generalization of DFT and the EFT is a generalization of GFT, for dynamic graphs. We have added discussions of this in the revised version of the paper.

Reviewer comment (Theory):
What will be harmonics for ring of ring graphs (no edge addition or dropping over time)?

Author response:
- For a ring graph we can show that the laplacian is a circulant matrix and the eigendecomposition gives the basis vectors of the DFT(classical Fourier transform) matrix with the eigenvalues the corresponding frequencies. For a ring of ring graphs, we can show (using property 1) that the EFT matrix comes out to be the Kronecker product of two DFT matrices (one along the time domain and one along the vertex domain). Similarly, the harmonics will be the Kronecker sum of the harmonics along the individual domains. To remark, we can say that the GFT is a generalization of DFT and the EFT is a generalization of GFT, for dynamic graphs. We have added discussions of this in the revised version of the paper.

Reviewer comment (Theory):
Despite giving guarantees on the approximation part, this work does not address the theoretical guarantees of learning under this module. Hence this makes this work more like heuristic. The reviewer believes that this might be quite difficult though.

Author response:
- The architecture is designed from the theoretical motivation. From the theory, we see how to decompose the joint Fourier Transform for the Dynamic graph into individual components of Fourier Transforms along the temporal aspect and vertex domain. The empirical approximation of the model further depends on the errors introduced by the components used to approximate the filters such as from the truncated Chebyshev polynomials etc. While we have shown from the empirical results of the neural architecture that learning the joint frequency components as proposed by the theory benefits the dynamic graph representation learning, it is an interesting point mentioned by the reviewer regarding deriving further bounds for the errors introduced by the approximation of the model implementation. 

We believe that, taking inspiration from previous works [1] we could work on further bounding the approximation ability of the model. Would you suggest we discuss this in the paper, especially as future work in the conclusion section?

Reviewer comment (Presentation):
The font size of the tables should be improved. This makes it hard to read. The reviewer suggests reducing presentation on the properties of EFT in proposition 5.1 to make up for the space and delay the properties of this transform to the appendix instead.

Author response:
- Thank you for your feedback on the presentation. We have incorporated the changes in the revised version of the paper. The new changes are marked with the green text color.

Reviewer comment (Experiments):
The question is how is the complexity of computing this transform. The reviewer did not see how fast it can be computed compared with other baselines and would like to see if it can be efficiently computed empirically.

Author response:
- The computational complexity of the transform is $\mathcal{O}(N^3T + NTlog(T))$. However this reduces in the neural architecture where we use polynomial approximations. The overall computational complexity of the neural architecture is $\mathcal{O}(E + NTlogT)$.
As suggested by the reviewer we also compute the wall clock running time of our method and compare it against baselines. The below table shows the time taken in seconds per epoch of our method (EFT) vs the average over the baselines. Our work significantly reduces the complexity at empirical level if simple eigenvector decomposition would have been considered against the proposed computationally efficient EFT. 

Dataset  | Average Baseline  | Full EVD | *EFT (our implementation)* | 
| :------------- | :----------: | -----------: | -----------: | 
|SBM| 68.35 | 1419.40  | 68.97 |
| UCI | 275.24 |  2493.20  | 276.36 |
| AS | 115.34 |   2614.59   | 116.60 | 
| Brain | 2.86 | 124.67  | 2.95  |
| Elliptic | 70.12 | 14329.54  | 71.66  |

From the above table we see that the running time of our method is comparable to the baselines and that empirically it can be computed efficiently. Note our implementation saves time over the full EVD version (which has a complexity cubic in the number of nodes).