Paper: Forked Diffusion for Conditional Graph Generation (ICLR-2024)

Reviewer comment (Experiments):
It is evident from recent studies [1,2] that both the genetic algorithm [3] and the reinvent method [5] serve as potent baselines. This paper should benchmark against a broader set of baselines to truly showcase its performance in molecular design tasks.

Author response:
Our method differs from GDSS as we are learning a joint dynamics between properties and structures. GDSS does not model properties. Differently from GDSS, our model is specifically designed for conditional tasks.

Reviewer comment (Theory):
The assumption that "the properties are independent conditioned on the structure" in Equation 15 is unjustified and not reliable.

Author response:
We have fixed this.

Reviewer comment (Experiments):
Is it possible to include more baselines like the DiGress [1] for comparisons?

Author response:
We added 3 baselines: U-bounds, L-bounds, \#Atoms, following [1,2,3].
The  upper bound ’U-bound’ baseline  randomly shuffles the property labels of molecules and reports their MAE. The ‘\#Atoms’ baseline predicts molecular properties based only on the number of atoms in molecules. Lower bound (L-bound) is the best performance bound from QM9.


Digress comparison is not appropriate as its conditional generation task, because
1) Digress does not compare to any previous works on conditional generation, such as EDM [1], EEGSDE [2], Jodo [3], all these methods provide comprehensive experimental details, and we compare with them. 
2) Digress does custom dataset processing, differently from the other benchmarks [1,2,3].

In our first experiment (Section 4.1) we follow the setup of GDSS [4], in Section 4.2 we follow the setup from EDM [1], EEGSDE[3], Jodo[3].

In addition, Digress codes presents reproducibility issues for the conditional generation, as mentioned by the authors on github: "Warning: The paper experiments were run with an old version of the code. We have incorporated the changes into the public version to create this branch, but we have not tested thoroughly yet. Please tell us if you find any bugs." The problem is confirmed by various issues raised on github \url{https://github.com/cvignac/DiGress/issues/68}. 

This said, we can still try if the reviewer think necessary, and we added digress to related work.

Reviewer comment (Evaluation):
Some molecular structures in Figure 2 do not appear reasonable. Could the authors provide further analysis?

Author response:
Could you elaborate please? According to our understanding, Figure 2 presents valid molecular structures.

Reviewer comment (Writing):
Explanation on key components lacks details: What do the structure variable ($x_s$) and child variables ($x_p$) actually represent during the experiments?

Author response:
The parent and child variables in each experiment vary, typically with the structure variable representing the graph and the child variables representing associated properties. Appendices B2, B3, and B4 provide detailed explanations for these variables.

Reviewer comment (Writing):
Explanation on key components lacks details: What does the models ($s_{\phi_i, t}$) approximate?

Author response:
$S_{\phi_i, t}$ serves as an approximation for the coupling between the $i^{th}$ child variable (or property) and the parent variable (graph). This influence on the reverse Stochastic Differential Equation (SDE) process enhances control over the conditional generation, allowing for a more nuanced and precise generation process.

Reviewer comment (Theory):
Explanation on key components lacks details: How is the training objective derived? Is it a straightforward extension of score matching objective?

Author response:
The training objective is based on score-matching but incorporates the forked diffusion settings given by propositions 1 and 2, which are novel results that we obtained via formal derivations in Appendix A.1 and A.2.

Reviewer comment (Theory):
Several claims made in the paper are not clear and require justification: (as explained in Intro) How does the parent process guide the child processes? From what I have understood, the parent process and child processes are dependent on each other rather than a specific process guiding others, similar to the processes of GDSS. In particular, the proposed diffusion framework seems to be a system of multiple processes (parent and childs) with some dependency conditions given between the variables, and the score functions are dependent on all the variables including additional context.

Author response:
A singular parent diffusion process is linked to a primary variable, such as the graph structure, while multiple child diffusion processes are dedicated to dependent variables, such as graph properties or labels. The parent process orchestrates the co-evolution of its child processes, as mathematically depicted by the system of equations governing the forward Stochastic Differential Equation (SDE) process in Eq. 2. Here, $y_{s}$ represents the parent variable (graph structure), and $y_{i}$ represents the dependent variable, i.e., properties. This influence persists in the reverse SDE, as illustrated in Eq. 4.

Reviewer comment (Theory):
Several claims made in the paper are not clear and require justification: Is the assumption on independence between child variables valid for the experiments? As I understand it, the child variables represent some chemical properties, which do not seem to be independent from each other.

Author response:
We adopt the assumption of independence among child properties as a modeling choice to formulate the forking process effectively. The consideration of chemical properties of molecules as either independent or dependent is contingent on the specific characteristics being examined. While certain chemical properties can be relatively independent, others exhibit interdependence, influenced by the underlying molecular structure. Leveraging this inductive bias, we incorporate this into our forked diffusion process to achieve a more nuanced control over the joint space of properties and molecules.

Reviewer comment (Experiments):
As FDP requires multiple score models (for the primary and child variables), I presume the number of model parameters required for FDP would be quite large. Ablation study on the number of model parameters could strengthen the effectiveness of the proposed framework.

Author response:
Could you let us know what kind of ablation study you require ? Is it like using all (n out of n) properties as children vs 1 out of n properties etc?

Reviewer comment (Reproducibility):
Important experimental details, for example, model architecture or what the variables actually represent, are not provided in the main paper. There seems to be some explanation in Appendix B, but it is not referenced in the main paper.

Author response:
We agree, will move such important details in the main paper.

Reviewer comment (Novelty):
Missed some previous works on conditional graph generation:  
- Vignac et al., DiGress: Discrete Denoising diffusion for graph generation, ICLR 2023  
- Lee et al., Exploring Chemical Space with Score-based Out-of-distribution Generation, ICML 2023

Author response:
Thanks for pointing out those relevant works. We will add into related work section.

Reviewer comment (Experiments):
Recent works (e.g., DiGress) find that using Graph Transformer architecture (instead of GNN-based architecture of GDSS) shows improved generation performance. Does FDP show similar improvement over GDSS when using the Transformer architecture?

Author response:
Q2: Thanks for the question. In fact, our implementations for are using transformers. With a careful read of the GDSS work [1] one can see that they are employing attention mechanisms. In fact, the architecture for processing graphs in GDSS is the graph multi-head attention (Baek et al.,2021) [2] . Our method in experiment 4.1 is built using similar architectures. For the experiment in 4.2 we compare to Jodo [3], which also uses Transformers, and we inherit a similar architecture for the our forked implementations for Section 4.2.

Reviewer comment (Novelty):
The motivation behind the proposal of forked diffusion is not clear in the introduction. While the categories of related work are presented, it is unclear what deficiencies these methods have compared to the proposed approach.

Author response:
We introduce an innovative method for refining guidance within diffusion models to achieve superior controllable generation. Our approach adopts a unique perspective by concurrently modeling the variable and its properties, leading to an aligned latent space. Empirically, our method outperforms naive conditional diffusion models guided by either classifier-free or classifier-based methods [5]. Furthermore, we establish a robust mathematical framework using Stochastic Differential Equations (SDEs) to derive both the forward and backward forking diffusion processes.

Reviewer comment (Novelty):
The related work is not comprehensive enough, as recent diffusion-based methods such as Digress [1], etc., have not been mentioned.

Author response:
Thank you for the reference. We have added Digress in Appendix C (extended related works), and will further refine, to bring it in related works. We put this reference appropriately in our paper.

Reviewer comment (Experiments):
In the conditional generation experiments, experiments on large-scale datasets and additional baselines for comparison should be included. Especially for the task of molecular generation, it would be beneficial for the authors to include recent methods such as DiGress [1], MiCaM [2], MolHF [3], etc.

Author response:
Thank you for providing the pertinent references for our research. We have incorporated these citations in appendix C. It's important to highlight that MolHF [3] does not possess a property conditional generation feature, thereby hindering a direct comparison with models that exhibit this capability.

While the MiCAM work [2] is related (included in the Expanded Related Works section in appendix C), it does not evaluate on Mean Absolute Error (MAE) like the literature we are following for property-conditional generation (e.g., [4,5]). In MiCAM [2], the task is goal-directed generation, making it not directly comparable with any diffusion approach considered in [4,5].

Reviewer comment (Presentation):
It is unclear why the molecular metrics results for the Zinc dataset are not presented in Table 3.

Author response:
Thanks for pointing our this issue. We are running the ZINC experiment and will complete that evaluation.

Reviewer comment (Novelty):
The proposed method is kind of a simple extension to current SDE diffusion models. There is not much technique improvement except introducing additional variables for these conditional properties inside the SDE process.

Author response:
The novelty of our approach is given by the following point:
We present an unexplored approach to characterize the propagation of conditional information, or guidance, within diffusion models, aiming for enhanced controllable generation. Our method introduces a novel perspective by jointly modeling the variable and its properties, resulting in an aligned latent space. Empirically, our approach outperforms naive conditional diffusion models, guided by classifier-free or classifier-based methods. Additionally, we establish a robust mathematical framework utilizing Stochastic Differential Equations (SDEs) to derive both the forward and backward forking diffusion processes.

Reviewer comment (Evaluation):
Most importantly, as the author just directly introducing these additional variables (with a simple conditional independent assumption among properties given the graph), the properties are not aligned with the intermediate graphs during the SDE process. This makes the method not very reasonable. For example, as there is no direct correspondance between graphs and properties, it is possible that the generated properties are not aligning with the generated graph. The alignment between property and graph at individual level is very important, all experiments don't have evaluation on this individual alignment. Only population-level MAE for these properties are evaluated.

Author response:
Our objective is to capture the joint dynamics between the graph (parent variable) and its associated properties (child variables) through our forked diffusion processes, both in the forward and reverse directions. In the inference phase, the reverse Stochastic Differential Equation (SDE) responsible for graph generation is influenced by the properties and their reverse process dynamics (refer to Eq. 5). This nuanced interplay allows for superior control over the generation process, showcasing exceptional performance compared to conventional methods relying on classifier-free or classifier-based guidance. It's worth noting that, like many molecule generation and property prediction methods in the Deep Learning literature [1,2], usually population-level MAE is used as a metric to quantify the performance of the generated molecules via the model which may result in some discrepancy towards the alignment. We follow the setups from such previous works [1,2], which have been published in top conferences (ICML,ICLR) and highly cited.

Reviewer comment (Evaluation):
In the experiment of generic graph generation, I don't see a clear difference of the population-level MAE between the proposed method and naive baseline GDSS, which may indicates that the method is not very effective.

Author response:
In the context of generic graph generation, we extracted graph properties from the data using the NetworkX library and benchmarked our approach against GDSS [3], one of the most recent state-of-the-art methods in this domain. Notably, our method demonstrates superior performance compared to existing alternatives. While these properties may not be as inherently intuitive as molecular ones, they serve as a testament to the efficacy of our approach, particularly on synthetic datasets.

Reviewer comment (Experiments):
Table 6 shows the result over unconditional generation, although this is not the intention of the designed method, I'm curious why the baseline only include GDSS-seq instead of the original GDSS.

Author response:
We agree that the unconditional tasks for our model did not show a strong result. Our method is mostly suitable for controlled generation, and will move this experiment into the appendix.

Reviewer comment (Presentation):
Figure 3, ego-small should be the right one?

Author response:
Yes, we will fix the typo.

Reviewer comment (Theory):
Can you summarize what you want to prove inside Appendix A.2? I'm kind of confused as I believe the reverse SDE formulation is a well-known result that is directly used.

Author response:
In Appendix A.2, we elucidate the reverse Stochastic Differential Equation (SDE) system governing our forked diffusion process. The derivation relies on the theoretical application of both forward and backward Kolmogorov equations to establish the reverse SDE for the joint system encompassing the graph and its associated properties.