Paper: Deep Graph Predictions using Dirac-Bianconi Graph Neural Networks (ICLR-2024)

Reviewer comment (Experiments):
Neither of the datasets considered include non-trivial edge features. Perhaps you could have flux along an edge in a powergrid or something?

Author response:
We experimented with using the power flow on the edges as additional input features, but it did not improve the performance. See Reply to all Reviewers - Further Datasets.

Reviewer comment (Experiments):
Not clear how the baseline methods were chosen. Are these also methods meant to address oversmoothing and long range dependencies? Are they at or near s-o-t-a?

Author response:
We have chosen the best performing models on the datasets as baselines, so we consider them state of the art. ARMANet and TAGNet, which perform best for the power grid task, share some features with architectures designed specifically to avoid oversmoothing, and are iterated quite deeply to achieve their performance. The DBGNN layer that beats them here is comparatively simpler, at least from a dynamics point of view.

Reviewer comment (Writing):
x' and e'_{i,j} are not defined in equations 6 and 7. I am assuming they mean the output of delta_{DB}(x, e), but this should be made more clear.

Author response:
Thank you for pointing this out, you are right, we will clarify this point.

Reviewer comment (Theory):
The leap from 8 to 9 should be made more clear. This appears to be solving \partial_t - \partial_{DB} \pm \beta = 0, or something similar, but this should be made explicit.

Author response:
Equations 8 and 9 together are the Bianconi's topological Dirac equation written out explicitly in coordinates. We are not solving anything, edges and nodes evolve at the same time. We will clarify this point.

Reviewer comment (Theory):
Is there any connection between the DB equation and the wave equation?

Author response:
This is an excellent question. The Dirac equation is a wave equation, however, it includes an overall factor $i$ in front of the right hand side. As we omit this factor, the equation we use is not directly a wave equation. Instead, we have exponentially growing and shrinking modes. We will clarify this point in the manuscript.

Reviewer comment (Theory):
Is there any way to use the DB operator to define Riesz transforms ($\partial_{x_i}\Delta^{-1/2}$ in the Euclidean setting)?

Author response:
This is an interesting question, we do not know. Generally speaking, Dirac operators tend to require internal spaces and do not directly operate on real valued functions. We expect that any connection would be subtle.

Reviewer comment (Theory):
Where does the network include non-linearities? Why do these not induce loss of energy?

Author response:
We include a ReLu after every step of the dynamics. It is unclear to us why a non-linearity should be expected to induce a loss of Dirichlet energy. We will update the section on Dirichlet energy to better represent typical conditions in the layer, in which case we do see a gradual decline, see Reply to all Reviewers - Dirichlet Energy.

Reviewer comment (Theory):
What is the computational complexity of DBGNN vs other methods?

Author response:
The time complexity of the forward pass is the same as that of Message Passing Neural Networks. The main difference to these, is the fact that we have the edges as full updating states of the layer iteration. This does not increase the computational complexity. We will include a small note in the manuscript to this effect.

Reviewer comment (Experiments):
The paper could be improved via more thorough experimentation and also analysis of / references to the theoretical properties of the DB operator.

Author response:
We will include a few more pointers to the literature on the boundary operator in the context of machine learning. The major point of our work is not the introduction of the boundary operator, but of Bianconi's Dirac equation, with dynamical edges and mass terms. We will clarify this point. We believe that Bianconi already provided a fairly thorough analysis of the dynamics of the Dirac equation on networks, but the addition of higher dimensional feature spaces will change the picture substantially.

We agree strongly that more experiments are needed to gain a better understanding of those layers properties, as well as the underlying dynamics, and will include more in the revised manuscript. See Reply to all Reviewers - Evidence for long range Capabilites and Further Datasets.

Reviewer comment (Experiments):
Limited ablation studies. It can be made further clear for how much performance gain comes from Dirac structure vs other enhancements.

Author response:
It is unclear to us what the reviewer views as important other enhancements. DBGNN is a relatively simple layer. It is the iterated application of equations (11) and (12), which are the direct analogies of the Dirac Bianconi equation, followed by a skip connection. We have added a direct comparison to a simple MPNN to clarify this, and we now show that the equivalent MPNN does not have comparable dynamics or long range feature propagation (Reply to all Reviewers - Evidence for long range Capabilities).

A more detailed study on how performance varies with various hyperparameters would, of course, be highly interesting. We also believe the layer to be highly compatible with the vast array of architectural enhancements that have been explored for MPNNs and GCNs. We think such broad questions are beyond the scope of this initial paper, though, and we decided to focus our experimental resources elsewhere.

Reviewer comment (Experiments):
The paper does not exploit edge features on binding affinity task. It may be unsure if Dirac structure helps here.

Author response:
It is important to note that in the absence of edge input features, we still have a non-trivial hidden edge dynamics. Thus, the way the nodal information propagates is shaped by the Dirac structure in this case, too. We will clarify this point in the paper.

Thus we consider the competitive performance on this task a strength of the layer. It shows that the underlying dynamics is interesting. We assume that useful edge features can only help to get higher performances. We are also exploring further datasets, see Reply to all Reviewers.

Reviewer comment (Experiments):
Although one of the experimental tasks is related to long range dependency, it is not sure for the second task. A robust study on some long range dataset benchmark would be required to verify holistically the claims of the model on long range tasks as is mentioned in multiple instances in the paper. The paper would be strengthened by a more in-depth analysis of what long-range effects the model is capturing in this domain.

Author response:
We agree that a much broader study would be needed. We have extended the analysis beyond the Dirichlet energy by including example trajectories of a randomly initialized layer. See Reply to all Reviewers - Evidence for long-range capabilities.

We also would like to point out that the power grid task is known to require long-range capabilities (Ringsquandl et al. (doi: 10.1145/3459637.3482464)), and that the state-of-the-art models (ARMANet and TAGNet) are quite deep. Together with the evidence from randomly initialized layers, we consider the conclusion reasonable that DBGNNs are good for long-range dependencies.

At the same time, we agree that a study of DBGNN on a long-range dataset benchmark would strengthen the paper considerably, and we are working towards that goal. See also Reply to all Reviewers - Further Datasets.

Reviewer comment (Experiments):
For binding affinity task, did you try using edge features? Does the Dirac structure help in that case?

Author response:
See Reply to all reviewers (Baselines for binding affinity).

Reviewer comment (Experiments):
The paper applies the DBGNN model to two vastly different tasks, power grid analysis and molecular property prediction, without tailoring the model for the specifics of each task. This lack of task-specific optimization could limit the model's performance and applicability.

Author response:
We agree that the performance of DBGNN could be further improved, but it already shows competitive and superior performance in comparison to the baselines. We consider it as a strength that the layer as presented is relatively simple but already exhibits interesting performance. Adding optimizations should therefore be possible, but we consider it beyond the scope of this paper, which focuses on the properties and performance of the core layer.

Reviewer comment (Novelty):
If I understand correctly, the Dirac-Bianci operator is called the boundary and co-boundary operators in the simplicial complex theory on graphs. Existing studies propose GNNs that use or extend them (e.g., [1--4]. Also, [5] does not use (co-)boundary operators but extends GNNs on simplicial complexes.) Therefore, it is debatable whether the proposed method has (theoretical and experimental) novelty and significance in terms of using the Dirac-Bianci operator.

Author response:
We thank the reviewer for highlighting the connection to this part of the literature. The comment was extremely helpful. We agree that the boundary operator in itself is not new, and we cite Loyd et al. as an early work on using the boundary operator in the context of topological data analysis on simplicial complexes.

The key point of Bianconi's Dirac equation is the combination of the boundary operator, and having the edge degree of freedoms as full participants in the dynamics with an appropriate mass term. This causes a spectral gap, and interesting propagation on the network. The work \[1\] on sheaf diffusion for example, while considering a very similar topological set up as we do, does not treat the edge space as possessing its own dynamics, and thus cannot induce analogous dynamics.

The topological set up is of the general form considered in \[2\], which we will note in the manuscript. However, in \[2\] the focus is on simplicial complexes, and in their applications to graphs, the set up we use is not considered (we have boundary and coboundary at the same time, but no up/down/Hodge Laplacians. The set ups of \[2\] did not feature the combination of a boundary/coboundary operator). In a further literature review based on the input of the Referee we identified ( arxiv.org/abs/2012.06010 ) as one example where a similar set up appears, and we will cite this as a predecessor.

Overall we believe our approach is closer to, e.g. the use of Kuramoto oscillators and other physically inspired dynamical equations that are not diffusive, rather than approaches inspired from algebraic topology. The close relationship to these works is of course highly relevant and of great theoretical interest. We will rework the theory section to make this clear.

Reviewer comment (Experiments):
This paper claimed that one of the advantages of the proposed method is that it solves over-smoothing. However, since there exist models that tackle over-smoothing, such as GCNII [6] and DRew [7], I have a question about whether the proposed method is superior to them.

Author response:
The motivation to explore the use of the DB equation was twofold, we wanted to be able to deal with edge features and to use a non-diffusive dynamical equation that would enable the deep predictions needed for power grid analysis. In our view, it is highly interesting that the DB equation possesses these qualities without having to be specifically engineered for it.

We agree that it would be interesting to understand in more detail how it compares to other approaches that were designed to handle oversmoothing. In the power grid context, deep GNNs such as ARMANets and TAGNets are known to have the best performance. These share some characteristics with GCNII. DBGNN is a relatively simple layer, that does not make use of these strategies, yet has competitive performance in this dataset at least.

It would be highly interesting to see whether these strategies, as well as those used in DRew, can be used to further improve the performance of DBGNN. We will add this as a research question to the outlook, but consider it out of scope for this paper.

Reviewer comment (Experiments):
How were the hyperparameters chosen? In particular, the numerical experiments use networks with 48 and 10 steps in total, respectively. However, it is yet to be known whether these hyperparameters are optimal. That is, a model with fewer steps might be sufficient. I suggest conducting ablation studies to see the sensitivity of performances to the number of steps.

Author response:
We conducted unstructured hyperparameter studies testing different number of steps. We believe that there is more potential when conducting more hyperparameter studies, but even with our small studies, we could already outperform current baselines.

Reviewer comment (Theory):
In Figure 5(a), the Dirichlet energy of untrained DBGNN is almost constant regardless of the number of steps. Is this expected from the theory?

Author response:
See Reply to all reviewers (Dirichlet Energy).

Reviewer comment (Evaluation):
In Table 2, it is shown that only 1 layer of DBGNN suffices. This is deviating from the story that stacking more layers helps in complex (hard) datasets.

Author response:
it is shown that only 1 layer of DBGNN suffices

This is probably a property of the dataset. If 1 layer with 12 steps already considers sufficiently large subgraphs, more layer will not be necessary.

Reviewer comment (Experiments):
I am willing to raise my score significantly if the story of the paper is in line with the findings and more insights with toy examples are presented.

Author response:
I am willing to raise my score significantly if the story of the paper is in line with the findings and more insights with toy examples are presented.

We are more than happy to provide further support for the story line of the paper. We will add a visualization of the long range response of the layer to a localized signal. See Reply to all Reviewers.  If the reviewer has further toy examples that they would find informative, we would be more than happy to implement them.

As the revision of the manuscript will still take some days we have already included the new visualization in a supplementary note.

Reviewer comment (Experiments):
Why do we need at least 13 layer GNNs for power grids? Is it true for all type of GNNs? Some powerful GNNs might need fewer than 13 layers.

Author response:
Why do we need at least 13 layer GNNs for power grids? Is it true for all type of GNNs? Some powerful GNNs might need fewer than 13 layers.

Ringsquandl et al. (doi: 10.1145/3459637.3482464) state that the unique structure of power grids leads to the need at least 13 layers for GNNs. Even that might not be sufficient for all tasks: the behavior of power grids shows profound non-locality. A failure in one part of the grid can cause a subsequent failure at a distance comparable to the system size (see e.g. <https://www.science.org/doi/10.1126/science.aan3184>). For example, large system splits and blackouts in Southern Europe have been caused by line failures in Northern Germany. The specific task we investigate is not known to be quite so non-local, but features the same physics.

Reviewer comment (Writing):
Please clarify using the notation $e_{ij}$ = $ - e_{ji}$.

Author response:
Please clarify using the notation e_ij = - e_ji .

See reply to all Reviewers - Edge notation.

Reviewer comment (Experiments):
What is the performance of the proposed method on node classification tasks on citation networks such as Cora, Pubmed, etc?

Author response:
What is the performance of the proposed method on node classification tasks on citation networks such as Cora, Pubmed, etc?

We do not expect that DBGNN has an advantage over existing methods on citation networks. Large topological patterns are not known to be relevant in this case, while close neighborhoods play a more significant role. See also Reply to All Reviewers - Further Datasets.

Reviewer comment (Evaluation):
The Dirichlet energy is constant over 500 steps in Figure 5? Why? Is it just for this experiment? It does not seem to change at all.

Author response:
The Dirichlet energy is constant over 500 steps in Figure 5? Why? Is it just for this experiment? It does not seem to change a t all..

See Reply to all reviewers - Constant Dirichlet energy.