Paper: Interpretable and Generalizable Graph Neural Networks via Subgraph Multilinear Extension (ICLR-2024)

Reviewer comment (Novelty):
The prediction performance of the proposed model does not see too much improvement over the state-of-the-art.

Author response:
We’d like to clarify that the generalization performance of the network also depends on the objective itself. In this work, we focus on the graph information bottleneck objective as GSAT[1] that can generalize under certain distribution shifts (i.e., FIIF shifts)[2]. 
- For real-world datasets, it’s unknown for the types of distribution shifts, thus the improvements to generalization performance may be limited.
- For spurious motif datasets, they are generated exactly following the FIIF shifts, hence **it’s expected that the graph information bottleneck shall hold stable and high generalization performance on spurious motif datasets**. From the results it can be found that GSAT fails to do so as GSAT can not reliably approximate SubMT, while GMT yields a stable and good generalization performance, improving GSAT up to 10% in prediction performance.

Reviewer comment (Theory):
Page 4: "which implicitly assumes the random graph data model (Erdos & Renyi, 1984). Def. 3.1 can also be generalized to other graph models with the corresponding parameterization ... ". Can you please comment on this further if this admits such a straightforward generalization to other models than ER, such SBMs or graphons as you suggested?

Author response:
One of the key differences between ER and the other two graph models SBMs and Graphons is that, the latter considers the generation of the graph in a blockwise manner. Within each block, the edges have a similar probability of occurring among nodes in the block. The subgraph multilinear can therefore be extended to a hierarchical variant to adhere to the blockwise property.

Reviewer comment (Writing):
What is $k$ in Eq. 9? Does it mean an application of k-layers of linear message-passing layers? In continuation, in Eq 11, what is the intuition behind the Hadamard operation between $\hat{A}$ and $A^{k-1}$? Section 5.1 needs to be written more clearly.

Author response:
Yes, $k$ in Eq 9 refers to the $k$ layers of message passing. We have supplemented the explanation to Eq 9 in our revised version.

As the non-linearity of $k>1$ weighted message passing in Eq 9 is the main cause for the approximation failure, the key idea for adopting the Hardamoard operation is to inject more “linearity” in the message passing by reducing the weighted message passing rounds to $1$. We revised Section 5.1 accordingly to improve its clarity.

Reviewer comment (Theory):
For Proposition 3.3, in my understanding, it basically says with > 2-layer linear GNN, Eq (6) cannot approximate SubMT. Do you provide any proof of that? Besides, I am curious if we have any empirical findings regarding this.

Author response:
Thank you for pointing out the missed proof of Proposition 3.3. We have supplemented the proof in the revised version. The idea to prove for linear GNN is intuitive:
a. First, one could write the outcomes of $f\_c(A)=\rho(A^kXW)$ as a summation in $A^k\_{i,j}v\_j$, with $v\_j$ is the j-th entry of the vector $v=XW$. 

b. Given the linearity of expectations, the comparison between $E[f_c(A)]$ and $f_c(E[A])$ now turns into the comparison between $E[A^k_{i,j}v_j]$ and $(E[A_{i,j}])^kv_j$. 

c. Since $A_ij$ is drawn from the Bernoulli distribution, with the expectation as $\widehat{A}\_{i,j}$, it suffices to know that $E[A^k\_{i,j}v\_j]=1^k\widehat{A}_{i,j}+0^k(1-\widehat{A}\_{i,j})=\widehat{A}\_{i,j}$, while $(E[A\_{i,j}])^k=\widehat{A}\_{i,j}^k$. Then, we know that $E[f\_c(A)] \neq f\_c(E[A])$.

> W1.2. Do you provide any empirical findings for Proposition 3.3?

Yes. We have conducted experiments to measure the total variation distance between the simulated SubMT, i.e., $E[f_c(A)]$, and the previous weighted message passing, i.e., $f_c(E[A])$.  Shown as in Fig. 14 to 17 in the revised version, we evaluate the SubMT approximation gaps of GSAT implemented in GIN and SGC on BA\_2Motifs and Mutag respectively. To fully verify Proposition 3.3, we range the number of layers of GIN and SGC from 1 to 5. It can be found that the results are well aligned with Proposition 3.3:
- When the number of layers is 1, the SubMT approximation gap is generically the smallest, because of more ``linearity'' in the network.
-  While along with the growing number of GNN layers, the network becomes more ``unlinear'' such that the SubMT approximation gap will be larger.

Reviewer comment (Presentation):
Table 4 is before Table 2 and 3. It would be better to fix this.

Author response:
> Q1. Positions of Table 2-4.

We have fixed it in the revised version.