Paper: Molecule Generation by Heterophilious Triple Flows (ICLR-2024)

Reviewer comment (Novelty):
Researchers already find that heterophily is not always harmful and homophily assumption is not always necessary for GNNs [1,2,3,4]. How does this paper align with these works?

Author response:
**Q1:** *There are many literatures showing that heterophily is not always harmful. How does my paper align with these?*  
**A1:** We thank the reviewer for pointing out these works. We are amending the related work section to include these works. We summarize the relation to these works as follows:

* Paper [1] points when the cross-class neighbour similarity is stronger than intra-class similarity, which keeps the post-aggregated embeddings of different classes distinguishable, then the data heterophily is not that harmful for node classification. 
* Paper [2] claims their newly designed aggregated homophily is a good indicator of the model expressiveness. However, it’s not very surprising that this homophily indicator is highly correlated with the performance of the model on certain data, since it combines the information from the data (node features, aggregation operator and labels) and model structure (assumption on model type: simplifying graph convolutional networks). In the model design, we share a similar idea: trying to process lower/higher frequency information separately in different channels. At the same time, we are different: For example, while handling high-frequency info, we avoid the information exchange between highly different nodes but keep smoothing the similar clusters. ACM capture high-frequency data by the graph Laplacian, which cares more about the graph topology (this idea comes from graph signal processing)
* Paper [3] verifies that compared with homophily, GNN performance on node classification is more related to intra- and inter-class Node Distinguishability (ND). In molecular generation, graph distinguishability is much more important than ND.
* Paper [4] defines the majority pattern (homophilic nodes in homophilic graph/ heterophilic nodes in the heterophilic graph) and shows that GNN is superior to graph-agnostic models on the majority patterns but worse on minority patterns. It tries to reveal how does the heterophily influence the GNN performance on node classification.

In general, all these papers [1,2,3,4]:
* Focus on the deeper insights between homophily and the GNN model performance on the node classification task. The heterophily matters in different styles: the difference with node classification and graph generation is discussed at the general reply.
* They try to discover the more essential factor (compared with heterophily) which affects the model performance. At the same time, we develop our methods, motivated by noticing the possible lossing information during training from the inspiration of heterophily.

Reviewer comment (Experiments):
Heterophily was usually studied in node classification tasks when graph-aware models underperform graph-agnostic models. I'm not sure if heterophily will also cause performance degradation of graph-aware models in generative tasks. Do you have any evidence or references? If the answer is yes, it would be a good contribution.

Author response:
**Q2:**  What is the problem of data heterophily in molecular generation tasks? any evidence or reference?
**A2:** Key to the generative model is learning the data distribution, which possibly contains special patterns underlying the data domain. Here the data is graph-structured, so an expressive model can distinguish different graphs. The smoothing embeddings from the message passing scheme will decrease the expressiveness of models. Thus it is natural to build structure to mitigate this problem. This is the motivation for us to design different flows to capture more information and avoid missing certain types of information. We have included an example analysis in the general reply to all reviewers.

Reviewer comment (Writing):
In equation (2), what is S?

Author response:
**Q3:** In equation (2), what is $S$?  
**A3:** In the ACLs (affine coupling layers), the input $X$ is split to two parts $X_1, X_2$ by the masking matrix. One part $X_1$ is updated by another part $X_2$ in the following style. For reversibility, the transformation is chosen to be an affine map, then $X_1$ is updated by the factorization matrix $S$ and translation matrix $T$ to be $SX_1+T$. And $S, T$ are generated from $f(X_1)$, where the $f$ is the coupling function. To avoid the negative factor, we choose $\log S, T = f(X_1)$, as mentioned in Equation (2). The coupling function $f$ is defined to be the GNN with heterophilous message passing layer in Equation (9): $ACL^{(GNN_i^\gamma, \mathbf{M}_i)}$

Reviewer comment (Experiments):
Ablation study is missing. How does each component in heterophilious message passing contribute to the performance gain?

Author response:
**Q4:** *Ablation study is missing. How does each component in heterophilious message passing contribute to the performance gain?*  
**A4:** Our model adds structure caring about heterophily/homophily based on MoFlow. And MoFlow utilizes the traditional GNN so loses the high-frequency information on heterophilic data. Thus MoFlow could be viewed as a single flow (HTFlows without heterophily and homophilic flows) as ablation. And there is also an example analysis in the general reply.

Reviewer comment (Theory):
The level of homophily in a graph is defined based on node labels in GNN literature, where high homophily is observed when neighboring nodes share the same labels, and vice versa. However, in the design of the paper’s heterophilious message passing layer, i.e., equation (6), they define the homophily of nodes as the cosine similarity between pair node embeddings. This could be problematic since the cosine similarity between pair node embeddings might not align with their labels. As a result, the proposed model may inherently fail to work well in the cases that the cosine similarity between pair node embeddings is not aligned with their labels.

Author response:
**A1:** This inconsistency appears in the node classification task as discussed in the general reply. Here the node features are the one-hot encoding of the node label (atom type). Since the node label is the learning target during the molecule generation, we need to design a substitute way to approximate the homophily/heterophily. The flow maps the sampled Gaussian embeddings back to the one-hot encoding of atom type (label), to be consistent with the definition of homophily of the final layer, we extend this concept to the intermediate layers as the cosine similarity between the node embeddings.

1. The node features $x_v$ of node $v$ is initialized by the one-hot encoding of atom type. we can write this to be  this node 
$$ x_v = [1_{v=C}, 1_{v=N}, 1_{v=O}, 1_{v=F} ] $$
e.g. for atom C, the node feature is $[1,0,0,0]$.

2. In the flow, the node features are mapped into latent space with a Gaussian distribution. During the molecular generation process, we sample Gaussian distributed random vectors and map them back to node features at the final layer, then this feature is transferred to a node label. It means we don't know the node label until the final layer. To define the homophily for a node in the intermediate layer without a label, we choose the similarity between the half-transferred embeddings to be the homophily. This definition style is also consistent with the final layer.

Reviewer comment (Novelty):
Numerous GNN architectures have been developed for heterophilic graphs, where they have demonstrated that their effectiveness for heterophilious graphs comes from their ability to work as high-pass filters. However, there is no solid justification indicating that the proposed heterophilic flows can effectively handle heterophilic graphs. Additionally, it is also not clear what the benefits of the designed heterophilic flows are when compared to directly adapting existing heterophilic GNN structures for molecular generalization. More discussion here would be helpful.

Author response:
**A2.1:** This is not a node classification task but a generation task. It could be easy to show the ratio of the heterophilic nodes is correctly classified. But here for the generation task, the model is supposed to learn the data distribution, with as least missing information as possible. Traditional GNNs will smooth the high-frequency information on the graph easily. We provide example analysis and discussion in the general answer. 

**A2.2:** The difference between node classification and graph generation task are discussed at general reply. For molecular generation task, graph distribution learning requires avoiding any possible information loss. The existing heterophilic GNN could mitigate oversmoothing problem for node classification. The heterophilic GNNs are good at making heterophilic nodes distinguishable. But it's not enough for simply replacing traditional GNN by it in current problem scenorias, because it may ignores the subtle low-frequency in the homophilic data. Keeping both traditional GNN (in central flow) and extra two flows helps the model get distinguishable embeddings (from both low and high frequency) for all types of graph-structured data points.

Reviewer comment (Experiments):
Lack of ablation studies on different components. The proposed model consists of several components, including bond flow and heterophilious atom flow. Moreover, the heterophilic atom flow encompasses three interacting flows: the central, homophilic, and heterophilic flows. It remains unclear which component is most crucial or how each contributes to the model's overall performance. A detailed breakdown and analysis would provide greater clarity.

Author response:
**A3:** Our model adds structure caring about heterophily/homophily based on MoFlow. And MoFlow utilizes the traditional GNN so loses the high-frequency information on heterophilic data. Thus MoFlow could be viewed as a single flow (HTFLows without heterophily and homophilic flows) for ablation study.

Reviewer comment (Novelty):
Given that this study seeks to address the challenges of heterophily and oversmoothing in GNNs for molecular generalization, it would be beneficial to delve deeper into papers on GNNs concerning heterophilious graphs and oversmoothing problems in the related work section.

Author response:
**A5:** Following reviewer **VenB**'s comments, we are amending the related work section with more details and background on heterophily.

Reviewer comment (Evaluation):
The proposed HTFlows only brings improvements on limited metrics when compared to state-of-the-art baselines (as listed in Table A4, A5, and Table 3), which constrains its contribution and impact.

Author response:
We focus on showing the benefits of our HTFlows framework over MoFlow, which can be seen as a base. Rather than cherry-picking just a few metrics where we excel, we on purpose show a wide range of chemoinformatics metrics. Some of the metrics are a bit of a trade-off; improvement on one metric can have a negative influence on other metrics. For example, if a model would always generate molecules with only carbons, it will get 100% validity, but the diversity should be low. Especially on QM9 (that other works typically test on), we achieve very good performance overall.

Reviewer comment (Evaluation):
It is unclear about the connections between the improved metrics and the introduced heterophilious triple flows.

Author response:
Generative model aims to learn the distributions related to several important patterns from the dataset. Most GNN-based models fail to capture enough information from the heterophilic graph data since the message-passing scheme smooths the graph embeddings after several layers. These improved metrics are good indicators showing the better distribution learning of our model.

Reviewer comment (Theory):
Why the mixing of ACL in the heterophilous atom flow are performed one by one for each atom type rather than in parallel?

Author response:
The different flows learn different types of information. We design the interactions for different channel so that they could also utilize the information from other patterns. However, for the reversibility, ACL updates one part of data by another part, which means there is always an order for which part is updated (impossible to update all at the same time, otherwise the structure is not reversible anymore). More layers of the current design can also make sure enough interactions compared to parallel design.

Reviewer comment (Evaluation):
It seems that the histograms in Fig. 6 and 7 do not match the results in Table A4 and A5. For example, why the molecular weights in Fig.6 are mainly 75~175, but the mean molecular weights in Table A4 are much smaller? Besides, as shown in Figure 7, the molecular weights of GraphDF seem to have a significantly lower mean value than MoFlow and HTFlows based on their histograms, but HTFlows has the lowest mean molecular weight as listed in Table A5.

Author response:
They are consistent. As described in App. C, the metrics in Table A4–A5 are the Wasserstein distance/discrepancy metric between the generated molecules and the original data set, summarizing the histogram result into one number.