Paper: DIRECTIONALITY IN GRAPH TRANSFORMERS (ICLR-2024)

Reviewer comment (Theory):
Although computational limitation of the proposed architecture is discussed briefly, the runtime complexity is quadratic in number of nodes like vanilla transformers. If sparsity helps, how would the complexity differ for DiGT compared to other non-directional graph transformers?

Author response:
The complexity of graph transformers typically depends on their structural design. Standard graph transformers, such as Vanilla Graph Transformer and DiGT, assume that all nodes are interconnected, resulting in a complexity that is quadratic relative to the number of nodes. In contrast, graph transformers employing sparse attention focus only on neighboring nodes and ignore non-existent edges. This approach reduces both computational time and memory requirements and works like graph neural networks. We will explore sparse attention techniques in the future.

Reviewer comment (Evaluation):
Are there other ways to quantify directionality of graphs besides SCC entropy?

Author response:
There are no definitive measures to characterize the importance of directionality. One option is to use various directed graph metrics like distribution of in and out-degrees, directed cycles, etc. Another option is to use some spectral measures. We used the SCC entropy that combines some aspects of the directed metrics, and we also study the empirical effect of random edge flipping. We believe this is an interesting open question for the graph learning community.

Reviewer comment (Novelty):
A closely related work "Edge Directionality Improves Learning on Heterophilic Graphs", Rossi et al., 2023, is not discussed in this work which would be important and provides (i) necessary context on novelty (ii) applicability of DirGNN on vanilla GTs.

Author response:
Thank you for suggesting the Dir-GNN work, which is a directed graph neural network that focuses on incoming and outgoing edges. However, DiGT assumes that all the nodes are connected since our model is a graph-level transformer model. Dir-GNN was first published on May 17, 2023, making it concurrent to our research. We will thoroughly explore it in our future studies.

Reviewer comment (Theory):
[W1] **There are some questionable design choices in DiGT that seemingly contradict with the overall direction, yet are missing additional explanations.** In particular, the end of Section 2 mentions how directed message-passing GNNs "suffer from convolutional inductive bias... restricted to only the given neighborhood structure", yet DiGT uses attention that is localized to the k-hop neighborhood. Doesn't this essentially downplay the advantage GTs have over message-passing GNNs?

Author response:
We choose to retain the k-hop constraint, since it helps somewhat, as shown in Table 11 in the appendix. We replicate the last two rows of that table here:

| Model              | Flow2 | Flow3 | Flow6 | Twitter3 | Twitter5 |
|--------------------|-------|-------|-------|----------|----------|
|DiGT-DA (no k-hops) | 96.52 | 74.33 | 46.22 | 91.31    | 85.46    |
|DiGT                | 97.42 | 74.55 | 46.80 | 91.67    | 85.94    |

As we can see the use of k-hops confers a slight benefit. But, one critical difference from GCNs should be noted. The GCNs are always constrained by the direct neighbors which add a very strict inductive bias. On the other hand, when we used k-hops, we treat the entire set of k-hop neighbors as the context, i.e., any node can directly communicate with any other node *within* the k-hops, which is a much less restrictive "bias", and lets us tradeoff the global attention with somewhat localized attention. 

When we say that we use dual encodings throughout, it does not imply that there is no exchange between these source and target embeddings. In fact, it is clearly of benefit to allow information to flow from one to the other, enabling them to mutually exchange information.
We do not want to  reduce the system to a mere duplication of single-encoding attention, so it is not clear whether the reviewer is suggesting that keeping them independent is a better choice?

Reviewer comment (Theory):
Also, the end of Section 3 mentions that "DiGT uses dual node embeddings in all layers", but Equations (9) and (10) imply that dual embeddings do not remain dual throughout the encoder, but are instead merged together into a single embedding $\mathbf{Y} \in \mathbb{R}^{n \times d_p}$, then separated via linear layers $L_{VS}$ and $L_{VT}$ once every layer. Why is this the case?

Author response:
When we say that we use dual encodings throughout, it does not imply that there is no exchange between these source and target embeddings. In fact, it is clearly of benefit to allow information to flow from one to the other, enabling them to mutually exchange information.
We do not want to  reduce the system to a mere duplication of single-encoding attention, so it is not clear whether the reviewer is suggesting that keeping them independent is a better choice?

Reviewer comment (Experiments):
The pipeline used to generate the Twitter datasets also seems problematic, due to how the label of each graph is chosen. Specifically, Twitter5 has 5 labels, each corresponding to the perturbation rate in [0, 25, 50, 75, 100]% used to rewire or reverse each edge in the original ego-network. Then for the experiments in Table 1, randomly flipping 50% of the edges in a Twitter5 graph labeled 1 (originally perturbed by 25%) would return a graph that is equivalent to a graph labeled 3 (originally perturbed by 75%) with no edge-flipping. In essence, it is unclear whether the drop in performance shown in Table 1 is simply due to having noisy labels, rather than the dataset exhibiting significance on directionality.

Author response:
Since the study of flipping edges is an independent experiment, we establish a standard pipeline applicable to all datasets. In the case of the Twitter dataset we first generate Twitter datasets for all the experiments. Then, for the random flip study, we randomly flip the edges for each step. The changes in the accuracy highlight the impact of edge directionality in this dataset, though there may also be an effect of class label confusion. Please note that, whatever may be the final effect, DiGT displays superior performance compared to all other (undirected) models, which leads further credence to the important role of directionality in the Twitter dataset.

Reviewer comment (Experiments):
[Q2] **Missing results for Malnet-tiny of Tables 2 and 3.** Are these all blank due to out-of-memory issues? If so, it would be better to fill them in with OOM.

Author response:
Thank you for your suggestions. We will mark "OOM" on these tables.

Reviewer comment (Theory):
Some designs are not well justified by either theoretical or empirical evidence. For example, the design of the positional encoding. No illustration, for instance, unidirectional counterparts or theoretical justification, has been provided about why the design is suitable for the directional situation.

Author response:
We used SVD-based positional encodings (PEs) since the left and right singular values/vectors can be used very naturally for directed graphs, unlike the Graph Laplacian, which is symmetric. For a comparison with Magnetic Laplacian PEs, see Q2 below.

Reviewer comment (Experiments):
The computational complexity of the proposed model appears high. It may have scalability issues when applied to large datasets.

Author response:
We have included the number of parameters used by our model and others in the general response above. Thus, our method is no more and no less scalable than other (dense) transformer based models.

Reviewer comment (Experiments):
Would it be possible to include some node-level experiments to further examine the capability of the model? There are a few commonly used benchmark datasets with directed graphs for node level tasks, e.g., Actor. Are such graph datasets too large for the proposed model to handle?

Author response:
Q1. We ran our model on the WebKB and the Telegram datasets. WebKB contains Cornell, Wisconsin, and Texas datasets. The table below reports on the accuracy of DiGT vs other baselines; we see clear improvements using DiGT (we do not include std-dev for clarity, but we will include them in our revised paper). Nevertheless, transformer-based models work only for medium sized datasets; for large number of nodes all (dense) attention methods will have scalability issues.  

| Datasets    | Cornell | Wisconsin | Texas | Telegram |
|-------------|---------|-----------|-------|----------|
| DGCN [ii]   | 65.13   | 71.08     | 64.90 | 85.77    |
| DiGCN [i]   | 45.68   | 50.54     | 53.73 | 65.96    |
| DiGCNIB [i] | 41.08   | 50.81     | 56.47 | 56.35    |
| EGT         | 71.43   | 73.59     | 81.18 | 80.00    |
| DiGT        | 71.62   | 74.05     | 83.73 | 90.38    |

[i] Z. Tong, Y. Liang, C. Sun, X. Li, D. Rosenblum, and A. Lim. Digraph inception convolutional networks. Advances in Neural Information Processing Systems, 33, 2020.  

[ii] Z. Tong, Y. Liang, C. Sun, D. S. Rosenblum, and A. Lim. Directed graph convolutional network. arXiv preprint arXiv:2004.13970, 202

Reviewer comment (Experiments):
Based on the recent literature, [1] also presents a positional encoding design tailored for directed graph Transformers. It would be valuable to see a comparative evaluation of your proposed model with this work.

Author response:
Q2. Thanks for suggesting the use of the Magnetic Laplacian PEs, which are indeed even better than SVD based positional encodings, as shown in the table below (we will include these new results in Table 2 in our updated paper; we do not show the std-dev values below for clarity, but we will include those as well):  

| Datasets | Flow2 | Flow3 | Flow6 | Twitter3 | Twitter5 |
|----------|-------|-------|-------|----------|----------|
| DiGT-SVD | 97.42 | 74.55 | 46.80 | 91.67    | 85.94    |
| DiGT-Mag | 97.00 | 74.92 | 47.49 | 93.34    | 86.61    |  

Here DiGT-Mag is with the magnetic Laplacian position encodings.

Reviewer comment (Experiments):
In the experiments, the author mentions abandoning the use of some available datasets because they think the directionality of these datasets is unimportant. I don't entirely agree with this viewpoint. I think that even if directionality is not crucial, if a model can encode directionality, it should still yield some benefits.

Author response:
We disagree. If a dataset has no directionality, then how or why will a directional transformer help? For example, if a graph is undirected, then its adjacency matrix A, will be symmetric, and the notion of hub or authority collapse -- $A^TA$ and $A A^T$ will be the same.

Reviewer comment (Experiments):
Some important baselines were not compared in the experiments. For example, the method proposed in the paper "Transformers Meet Directed Graphs [1]" achieved the SOTA result on Ogbg-Code2.

Author response:
Please note that directionality is not significant in the original ogb-code2 dataset, as we have shown in Table 1. Furthermore, please note that Geisler et al preprocess and modify the original graph to get competitive results, but we consider the original graph, which has little directionality.