Paper: HoloNets: Spectral Convolutions do extend to Directed Graphs (ICLR-2024)

Reviewer comment (Reproducibility):
The formulation seems to be restricted to one-dimensional node weights for the nodes, but it is not clear what modifications should be made if we are given multiple-dimensional node features.

Author response:
Channel mixing in spectral graph networks is implemented in analogy to the Euclidean case: A Euclidean convolutional filter applied to an image acts only on a single (r, g or b)-channel. Channel mixing is then facilitated by connecting all channels in two consecutive layers via convolutional filters. All incoming information at a given channel is then summed up. A convenient visual representation may e.g. be found in Figure 2 of (Koke (2023)) . 

Our method follows this paradigm. This might for example be explicitly seen in the layer update rule in Section 4: As described there, the  weight matrices $W_k^{\text{fwd/bwd},\ell+1}$ are of dimension $F_{\ell}\times F_{\ell+1}$, with $F_{\ell}, F_{\ell+1}$ the feature dimensions of the respective layers. 
These matrices implement the contact between different channels.

Reviewer comment (Theory):
The underlying reason for using holomorphic functional calculus has not been clearly demonstrated.

Author response:
The guiding principle behind modern spectral convolutional methods is that of designing filters of the form $g(T)$, with $T$ a characteristic operator and $g$ a learnable (scalar) function (Defferrard et. al (2016)). In the undirected setting, we had $T = T^*$, which – as we detail in Sections 1 through 3 – allows to unambiguously evaluate the scalar function $g$ on the matrix $T$ by essentially evaluating $g$ on each eigenvalue of $T$ independently.

As we detail in Sections 2 and 3, this is no longer possible in the directed setting; instead the only way to consistently define the matrix $g(T)$ is to make use of the holomorphic functional calculus. 

Following this concern raised by the reviewer, we have modified the corresponding discussion of this matter in Section 3.1,  to reflect even more strongly that $g(T)$ may only be consistently defined if $g$ is assumed to be holomorphic.

If the reviewer would however like us to incorporate additional specific comments on this matter, we would be more than happy to do so.

Reviewer comment (Writing):
Some typos exist. For example, in the "Contributions" part at the end of Sec. 1, there is one redundant "the" in line 6; in the second paragraph of Sec. 3.3.1, there is one more "near" in line 5; there should be an 'as' after 'referred to' right after equation (5) on page 17; the first 'eigenvalue' should indeed be 'eigenvector' in Sec. C on page 17; 'interpreting' should be 'interpreted' right after the mention of Appendix B on page 18.

Author response:
We are very grateful to the reviewer spotting these typos! We have dilligenly fixed them all in the updated version of our manuscript.

Reviewer comment (Theory):
The formulation seems to be restricted to one-dimensional node weights for the nodes. What modifications should be made if we are given multiple-dimensional node features?

Author response:
Channel mixing in spectral graph networks is implemented in analogy to the Euclidean case: A Euclidean convolutional filter applied to an image acts only on a single (r, g or b)-channel. Channel mixing is then facilitated by connecting all channels in two consecutive layers via convolutional filters. All incoming information at a given channel is then summed up. A convenient visual representation may e.g. be found in Figure 2 of (Koke (2023)) . 

Our method follows this paradigm. This might for example be explicitly seen in the layer update rule in Section 4: As described there, the  weight matrices $W_k^{\text{fwd/bwd},\ell+1}$ are of dimension $F_{\ell}\times F_{\ell+1}$, with $F_{\ell}, F_{\ell+1}$ the feature dimensions of the respective layers. 
These matrices implement the contact between different channels.

Reviewer comment (Theory):
What is the computational complexity of HoloNets compared to DiGCN, MagNet, and Dir-GNN?

Author response:
For definiteness, we will assume the DirGCN configuration of the DirGNN baseline (other configurations would yield higher complexity). For a given graph $G$, let $N$ denote its number of nodes and let $E$ denote its number of edges. Assume  we have input features of dimension $C$ and  a $F$ hidden dimensions in our architectures. 

DiGCN: In its full form, DiGCN has a $\mathcal{O}(N^2)$ complexity, which is reduced to a  $\mathcal{O}(EFC)$ complexity in its approximate propagation scheme, which is used in practice.

DirGNN: Similarly, DirGNN possesses a $\mathcal{O}(EFC)$. In both cases this stems from the sparse-dense matrix multiplications that facilitate the forward function.

MagNet: MagNet is similarly implemented via sparse-dense matrix multiplications; resulting in an $\mathcal{O}(EFC)$-complexity.

FaberNet: Since FaberNet too implements its forward using sparse-dense matrix multiplications, its filtering operation also has $\mathcal{O}(EFC)$ complexity.

Dir-ResolvNet: The filtering operation in Dir-ResolvNet is implemented as dense-dense matrix multiplication, and as such has complexity $\mathcal{O}(N^2)$ like the full (not approximated) forward operation of DiGCN would have.


Following this question, we have also included this discussion in a new section (Section I.3: Efficiency Analysis) in our updated manuscript.

Reviewer comment (Experiments):
It would be good to examine the efficiency of the proposed method, especially when compared with DiGCN and DirGNN. Moreover, a discussion on the number of learnable parameters would be beneficial.

Author response:
* It would be good to examine the efficiency of the proposed method, especially when compared with DiGCN and DirGNN.

We are very happy to do so:

For definiteness, we will assume the DirGCN configuration of the DirGNN baseline (other configurations would yield higher baseline-complexity). For a given graph $G$, let $N$ denote its number of nodes and let $E$ denote its number of edges. Assume  we have input features of dimension $C$ and   $F$ hidden dimensions in each layer of our architectures. 

DiGCN: In its full form, DiGCN has a $\mathcal{O}(N^2)$ complexity, which is reduced to a  $\mathcal{O}(EFC)$ complexity in its approximate propagation scheme, which is used in practice.

DirGNN: Similarly, DirGNN possesses a $\mathcal{O}(EFC)$ complexity. In both cases this stems from the sparse-dense matrix multiplications that facilitate the forward process.

FaberNet: Since FaberNet too implements its forward using sparse-dense matrix multiplications, its filtering operation also has $\mathcal{O}(EFC)$ complexity.

Dir-ResolvNet: The filtering operation in Dir-ResolvNet is implemented as dense-dense matrix multiplication, and as such has complexity $\mathcal{O}(N^2)$ like the full (not approximated) forward operation of DiGCN would have.

Following this point raised by the reviewer, we have now included this discussion into a new subsection titled “Efficiency Analysis” in "Appendix I: Additional Details  on Experiments” in our revised manuscript.
* "Moreover, a discussion on the number of learnable parameters would be beneficial."

Compared to DirGNN (in its DirGCN Setting) and assuming the same network width and depth, our FaberNet has $K$ times the number of learnable parameters in the real setting. In the complex setting, our method has $2\cdot K$ as many _real_ parameters as DirGNN.  Here $K$ refers to the highest utilized order of Faber polynomials $\Psi_k$. In our experiments, the maximal attained value of $K$ was $K = 5$ on  the “Squirrel” dataset. On this dataset, complex weights and biases performed best, so that our method has 6.3 M  trainable parameters in the corresponding utilized hyperparameter-configuration for Squirrel (detailed in Appendix I).

We have included this discussion above into the newly created subsection titled “Efficiency Analysis” in "Appendix I: Additional Details  on Experiments” in our revised manuscript.

Reviewer comment (Experiments):
The impact of the number of layers on the model's performance is worth exploring. Do we have the same problem of over-smoothing in directed graphs when the GNN goes deeper?

Author response:
* The impact of the number of layers on the model's performance is worth exploring. Do we have the same problem of over-smoothing in directed graphs when the GNN goes deeper?


We agree that this is an interesting future research direction. 

In our experiments, we already go up to layer-depth $6$, which is already deeper than the optimal depth for e.g. GCN. We will be glad to include a detailed ablation study on the number of layers in a camera ready version of our submission.

Here, we provide our theoretical opinion on this matter:

In the undirected setting, the degradation of feature-quality with layer depth is related to Laplacian smoothing. This smoothing operation makes node representations more and more similar as a signal is propagated through the layers (c.f. e.g. [1]). 

Mathematically, Laplacian smoothing can be understood as mapping an original feature matrix $X$ to its smoothed version $X \mapsto e^{-\Delta \cdot t}\cdot X$. Here $\Delta$ denotes a suitable graph Laplacian and the continuous time $t$ is replaced by the layer-depth $\ell$ when discussing the low-pass behaviour of GNNs. Smoothing now occurs, bevause the function $\lambda \mapsto e^{-\lambda\cdot t}$ suppresses all non-zero eigenvalues $\lambda$; especially as $t$ (or equivalently the layer-depth) is increased.  

For large times ($t \gg 1$) or respectively deep architectures, the only information that survives is that corresponding to the $\lambda = 0$ eigenvalue. Since the corresponding eigenvector is simply given (up to normalization) as $v_0^T = (1,…,1)^T$ (assuming that the graph is connected), this means that only a graph-average  $v_0^T\cdot X$ of all the available information $X$ survives. 

In the directed setting, it is possible that the undirected version of a directed graph is connected, while the original directed graph contains more than one _reach_ (c.f. the Discussion of reaches in Section 2 of our submission). In this case, there is no longer a single eigenvector associated to the eigenvalue $0$ of the (directed) graph Laplacian. Instead, the multiplicity of the eigenvalue $0$ is given by the number of distinct reaches (c.f. e.g. [2] ). Thus even in the infinite layer limit, atleast information about connectivity in terms of reaches will be retained. 

Additionally, we suspect that if some form of oversmoothing occurs, its “convergence speed” will be slower: Investigating  $e^{-\Delta \cdot t}$ from the spectral-response perspective introduced in our paper, we see that additional non-zero terms appear in the spectral decomposition of $e^{-\Delta \cdot t}$. These act counter to an overly-fast loss of information.

As a fully rigorous exploration of this topic would constitute a research paper in its own right, we leave further investigation of these ideas (beyond an additional empirical ablation studywhich we are performing and will include) for future research.


References

[1]: Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, Tarek Abdelzaher, Revisiting Over-smoothing in Deep GCNs, https://arxiv.org/abs/2003.13663

[2]: J. J. P. Veerman and Robert Lyons, A Primer on Laplacian Dynamics in Directed Graphs, Nonlinear Phenomena in Complex Systems,  2020, https://api.semanticscholar.org/CorpusID:211066395

Reviewer comment (Evaluation):
Did HoloNet outperform the baselines in graph classification tasks? More evaluation tasks could be investigated.

Author response:
* More evaluation tasks could be investigated.

In principle, we agree that more evaluation tasks could potentially be investigated. However, in an effort to not overload the (in our opinion) already quite information-rich paper, we limited ourselves to one task at the node-level and one task at the graph-level. 

Our aim in doing so is to showcase the potential, stability and expressivity  of directed spectral convolutional filters for both theseexperimental settings. We hope that this will result in a swift uptake of our HoloNet framework by the community and a further exploration of this research direction, with followup works on architectures e.g. adapted to additional tasks beyond the ones we already considered within our work.

Reviewer comment (Writing):
Although the technical insights shown in this paper are great and very interesting, the presentation of this paper, especially the main text, can be improved. The main goal of this paper, which is the implementation of the practical directed convolutional kernels (layers), should still be explained clearly in detail despite the authors' intention to convey as much information as possible within the main text.

Author response:
* "Although the technical insights shown in this paper are great and very interesting,  [...] the implementation of the practical directed convolutional kernels (layers), should still be explained clearly in detail."

We agree that our paper is written to serve a dual purpose: To --on the one hand-- develop the theory of spectral convolutions on directed graphs in detail so that it may be  taken up by the community for further research and on the other hand to establish our own specific architectures conforming to the developed framework.

Following this advice by the reviewer, we have now put additional emphasis on introducing and describing our specific architectures in more detail, as we explain below.

Reviewer comment (Reproducibility):
After reading the entire paper, it is still unclear whether it is necessary to perform the eigendecomposition of the graph operator. For example, even if we set the function to be Faber polynomials $\Psi_k(\lambda)=\lambda^k/2^k$, what will $\Psi_k(T)$ be in this case, where $T$ is the operator? Do we still have to consult Eq (3) to compute $\Psi_k(T)$, which can be computationally expensive? This type of question should be clearly explained in the main text to help readers better understand the big picture.

Author response:
* "For example, after reading the entire paper, I still do not know if we need to perform the eigendecomposition of the graph operator or not."

We thank the reviewer for this important and indeed central question.  The answer is that an explicit eigendecomposition (or in fact a Jordan Chevalley decomposition in the directed setting) NEVER needs to be computed in our approach. 

Following this question by the reviewer, we have now modified the final paragraph of the corresponding Section 3.2 in our updated manuscript. It now reads:
> “For us, the spectral response (4) provides guidance when considering scale-insensitive convolutional filters on directed graphs in Sections 3.3 and 4 below. The spectral response (4) is however never used to _implement_ filters: As discussed above, this is achieved much more economically via (3).”

* "[...] if we set the function to be faber polynomials $\Psi_k(\lambda) = \lambda^k/2^k$, what will $\Psi_k(T)$ in this case, where $T$ is the operator? "

If we have $\Psi_k(\lambda) = \lambda^k/2^k$ then we exactly have 

$$ \Psi_k(T) = \frac{1}{2^k}T^k.$$

We had initially discussed this effect of applying the holomorphic functional calculus to monomials and polynomials at the end of Section 3.1. Following this valid point raised by the reviewer, we have -- for additional clarity -- promoted said discussion at the end of Section 3.1 into a Theorem (namely Theorem 3.1 in our updated manuscript) into its own right for heightened visibility. 

Additionally, we now also explicitly provide the above result of applying the Faber polynomial $\Psi_k(\lambda)$ to the operator $T$ already in Section 3.3.1 (i.e. precisely when Faber Polynomials are first introduced).

* "Do we still have to consult Eq (3) to compute $\Psi_k(T)$, which can be computationally expensive? "

The answer to this question is a resounding NO:

Equation (3) never needs to be used to implement a filter in practice. The purpose of this equation is instead to serve as a theoretical tool to spectrally interpret the action of given filters. Additionally it provides guidance on the choice of suitable filters adapted to a given task.

To avoid a costly explicit Jordan-Chevalley decomposition, our model instead uses parametrized spectral convolutional filters, which were introduced and discussed in the first paragraph of Section 3.4.

This is indeed an important point brought up by the reviewer, and – as already mentioned above – we now explicitly state the following in our revised manuscript 
[Note that equation (4) in the revised document refers to equation (3) in the original submission]:
>“For us, the spectral response (4) provides guidance when considering scale-insensitive convolutional filters on directed graphs in Sections 3.3 and 4 below. The spectral response (4) is however never used to _implement_ filters: As discussed above, this is achieved much more economically via (3).”.

* "This type of question [as above] should be clearly explained in the main text to help readers better understand the big picture."

We hope that the provided modifications in our updated manuscript have clarified these points. Should this not yet be the case, we will of course be more that happy to act on any additional feedback.

Reviewer comment (Reproducibility):
Providing a pseudocode of the algorithm/training procedure could help clarify the computation process and address the above confusion.

Author response:
* "pseudocode of the algorithm/training procedure can help solve this issue."

Apart from the aforementioned added and modified discussions, we have also included a pseudocode of the forward of our model  in our revised manuscript (c.f. Appendix J) which we reference in the main text of our revised manuscript.

Reviewer comment (Theory):
There is a concern about the simulation of directed graph regression problems. It is not convincing why this should be treated as a directed graph problem, and how it helps when considering the edges to be directed and the nodes to be weighted. There may be physical or chemical interpretations, but this is left for other reviewers with domain expertise to decide.

Author response:
* "One minor concern that I have is about the simulation of directed graph regression problems. I am not very convinced why we should treat this as a directed graph problem, and how it helps when we consider the edges to be directed and the nodes to be weighted. Maybe it has some physical/chemical meanings, but since I am not an expert in physics, I will leave this to other reviewers to decide."

Our view of this matter is as follows: 

Our aim in this present paper is to establish within the graph learning community that the use of spectral methods on directed graphs is not limited to achieving  state of the art performance on node-classification: HoloNets can e.g. also be used at the graph level, to construct networks that are transferable between directed graphs describing the same underlying object at different resolution scales.

While transferability is certainly an established research topic for graph neural networks, the approach via of such multiscale-consistency has only recently started to be investigated. As such standard benchmarks have not yet been established; especially not in the directed setting. We thus modify a standard dataset (i.e. QM7) that was recently used to test for multiscale-consistency in the undirected setting (c.f. Koke et al. (2023)) in order to test for such consistency in the directed setting. 

Whether a molecule is represented as a directed or undirected graph is irrelevant from the perspective of retained physical information: Both descriptions may be translated into each other.
In either case, the node-weights arise naturally as a way to represent additive (under combining nodes) node-wise information, such as charges (like in the case at hand), particle numbers (e.g. Neutron- or Proton numbers) or masses corresponding to individual nodes/atoms.

Reviewer comment (Reproducibility):
From the implementation perspective, Dir-GNN is much simpler than FaberNet.

Author response:
* "I would not describe the difference between the performance of FaberNet and that of Dir-GNN to be significant, as most of the differences are within one standard deviation. Also from the implementation side, Dir-GNN is much simpler than FaberNet; "

That is a valid point. We have amended the corresponding statement about the comparison between Dir-GNN and Faber-Net in Section 5.1.

Reviewer comment (Writing):
On page 5, in the sentence "Faber polynomials provide near near mini-max polynomial approximation," there is a repetition of the word "near".

Author response:
* "on page 5, "Faber polynomials provide near near mini-max polynomial approximation" there are two "near"."

We thank the reviewer for spotting this typo, which we have now correced.

Reviewer comment (Reproducibility):
What is the choice of $y$ for Dir-ResolvNet?

Author response:
* "What is the choice of $y$  for Dir-ResolvNet?

We have chosen $y = -1$ for simplicity. We had already detailed this in “Appendix I: Additional Details on Experiments”, but have now also included this in the main text of our paper in our revised manuscript.

Reviewer comment (Theory):
Where do the imaginary weights in FaberNet come from? Is it because the eigenvalues can be complex?

Author response:
* "Where do the imaginary weights in FaberNet come from? Is it because the eigenvalues can be complex?"

Yes, that can indeed be thought of as the underlying reason. 

In more detail: If the underlying graph is directed, the associated characteristic operators $T$ are generically not self-adjoint. Hence eigenvalues of such operators are generically complex. If one intends to apply a function $g$ to such a matrix $T$, this necessitates $g$ to be defined at least in a neighbourhood of each eigenvalue of $T$, as can e.g. be seen from the spectral response discussed in the paper. Thus $g(z)$ needs to be defined for complex $z$. If one represents such a function via simpler functions (e.g. a polynomial $g(z)$ represented via a sum of monomials $z^k$ as $g(z) = \sum_k a_k z^k$ ), the corresponding coefficients $a_k$ are generically complex. These coefficients precisely constitute the learnable parameters in our method.