Paper: Streamlining Generative Models for Structure-Based Drug Design (ICLR-2024)

Reviewer comment (Novelty):
The method did not actually achieve de-novo drug design, as it sampled the unlabeled ligand graph from an existing dataset, raising questions about the novelty of the generated drug.

Author response:
*Firstly, it is important to note that the method did not actually achieve de-novo drug design, as it sampled the unlabeled ligand graph from an existing dataset. This raises questions about the novelty of the generated drug.*

This is a good observation. We would like to point out that we compare with other baselines also in terms of chemical diversity (main results Table 1) and obtain comparable results (68% vs 72%), where diversity is defined as 1 -tanimoto similarity as in all other baselines [1, 2].

**Novelty of the generated molecules.** The novelty of generated molecules with our method depends on which variant is used. The FastSBDD-$\mathcal{DR}$ described in Section 3.3.1 does not generate novel molecules, it is included to showcase methods capability for drug repuprosing. The other variants, however, FastSBDD-$\mathcal{ND}$ and FastSBDD-$\mathcal{PO}$ achieve 100% novelty, meaning that none of the generated molecules are present in the training sets. This shows that these two variants do in fact perform de-novo drug design.

Reviewer comment (Experiments):
There is a lack of clarity regarding whether bond types are preserved in the unlabeled graph. Even if bond types are not preserved, it is not intuitively clear whether there is enough flexibility for atom type when the graph structure is fixed.

Author response:
*Furthermore, there is a lack of clarity regarding whether bond types are preserved in the unlabeled graph. Even if bond types are not preserved, it is not intuitively clear whether there is enough flexibility for atom type when the graph structure is fixed.*

**Bond types are indeed preserved in the unlabeled graph.** We will make it more explicit in the paper. There is substantial flexibility for atom types with the fixed graph structure. We discuss in more detail below.

Reviewer comment (Evaluation):
The focus on overall diversity is not as meaningful as it could be. It would be more informative to assess and showcase the diversity of generated ligands for each template unlabeled graph, as this analysis can reveal the extent of chemical variation achieved by the method and is a more relevant measure of its potential impact.

Author response:
*In light of these concerns, the focus on overall diversity is not as meaningful as it could be. It would be more informative to assess and showcase the diversity of generated ligands for each template unlabeled graph. This analysis can reveal the extent of chemical variation achieved by the method, which is a more relevant measure of its potential impact. If the changes between the generated ligands and the template ligands are minor or do not significantly affect the chemical properties, the overall impact of the method may be considerably diminished.*

**Chemical diversity of our generated molecules is significant.** Thank you for pointing this out. There is in fact substantial variability in the chemical space even with fixed 2D unlabelled graph structures.

**New experiment based on your feedback.** In this experiment, we show that by only changing atom types, we can generate hundreds (and in 98% of cases at least a thousand) novel molecules, which are almost as diverse as the ZINC250k dataset itself. Please see the General Response for details.

Reviewer comment (Evaluation):
If the changes between the generated ligands and the template ligands are minor or do not significantly affect the chemical properties, the overall impact of the method may be considerably diminished.

Author response:
*In light of these concerns, the focus on overall diversity is not as meaningful as it could be. It would be more informative to assess and showcase the diversity of generated ligands for each template unlabeled graph. This analysis can reveal the extent of chemical variation achieved by the method, which is a more relevant measure of its potential impact. If the changes between the generated ligands and the template ligands are minor or do not significantly affect the chemical properties, the overall impact of the method may be considerably diminished.*

**Chemical diversity of our generated molecules is significant.** Thank you for pointing this out. There is in fact substantial variability in the chemical space even with fixed 2D unlabelled graph structures.

**New experiment based on your feedback.** In this experiment, we show that by only changing atom types, we can generate hundreds (and in 98% of cases at least a thousand) novel molecules, which are almost as diverse as the ZINC250k dataset itself. Please see the General Response for details.

Reviewer comment (Experiments):
Without more evidence and explanation regarding the mentioned issues, comparing the drug repurpose-like generating method with the de-novo generating method is not fair enough, and it might be more appropriate to compare it with other drug repurposing methods.

Author response:
*Without more evidence and explanation regarding the mentioned issues, I think comparing the drug repurpose-like generating method with the de-novo generating method is not fair enough. It might be more appropriate to compare it with other drug repurposing methods*

**Our scoring model enables both repurposing and new drug generation.** We note that the FastSBDD-$\mathcal{DR}$ variant of our method, designed for drug repurposing, was meant to demonstrate that our method can have wider applications. Due to our scoring model, we are able to perform both drug repurposing as well as novel drug generation with the latter being the primary concern. This is why we benchmark our method against the generative models for SBDD.

Reviewer comment (Reproducibility):
Is the bond type preserved in the template unlabeled graph?

Author response:
*Furthermore, there is a lack of clarity regarding whether bond types are preserved in the unlabeled graph. Even if bond types are not preserved, it is not intuitively clear whether there is enough flexibility for atom type when the graph structure is fixed.*

**Bond types are indeed preserved in the unlabeled graph.** We will make it more explicit in the paper. There is substantial flexibility for atom types with the fixed graph structure. We discuss in more detail below.

Reviewer comment (Evaluation):
Is the diversity of the generated ligands for a given template unlabeled graph good enough? Are they significantly different from the original ligand?

Author response:
*In light of these concerns, the focus on overall diversity is not as meaningful as it could be. It would be more informative to assess and showcase the diversity of generated ligands for each template unlabeled graph. This analysis can reveal the extent of chemical variation achieved by the method, which is a more relevant measure of its potential impact. If the changes between the generated ligands and the template ligands are minor or do not significantly affect the chemical properties, the overall impact of the method may be considerably diminished.*

**Chemical diversity of our generated molecules is significant.** Thank you for pointing this out. There is in fact substantial variability in the chemical space even with fixed 2D unlabelled graph structures.

**New experiment based on your feedback.** In this experiment, we show that by only changing atom types, we can generate hundreds (and in 98% of cases at least a thousand) novel molecules, which are almost as diverse as the ZINC250k dataset itself. Please see the General Response for details.

Reviewer comment (Theory):
The rationality of the two-stage method is suspectable. In general, especially from the perspective of structural biology, the protein-ligand interaction is strongly dependent on atom types, positions. However, in this paper, the only dependence on proteins is used to infer the unlabeled molecular graph where only topology information is used. And the second stage does not involve modelling the protein pockets. This contradicts with the fundamental knowledge of structural biology.

Author response:
This is a great point. We agree that this model definition and results are counter-intuitive and we believe that therefore it is especially important to publish such findings. Please see our General Response where we re-emphasize points made in the paper and discuss this model definition.

Reviewer comment (Experiments):
The main concern about the experiments is about the fairness of comparison with baselines. First, FastSBDD leverages additional molecular dataset, such as ZINC and ChemBL, while other baselines does not use additional datasets. Second, optimization is condered in FastSBDD. But the baselines are all generative methods. No optimization methods, such as MARS [1] and RGA [2], are included. From my perspective, it is more proper to compare FastSBDD with molecular optimization methods. RGA is a molecular optimization method proposed for SBDD.

Author response:
This is a good point. However, it is important to note two things. First, the additional datasets, such as ZINC and ChemBL only contain molecules, they do not contain molecule-protein pairs indicating strong binding of a given molecule to a given protein pocket as in the CrossDocked dataset used by SBDD
methods.

Second, our method is conceptually different than other benchmarks. Our approach with utilizing additional molecular datasets is novel in the SBDD domain and a part of our contribution. It is unclear how other baselines could benefit from these additional data.

Thank you for pointing this out. Please see our general response, where we discuss comparison with optimization-based methods.

Reviewer comment (Experiments):
Many important SBDD baselines are missing, such as DecmopDiff [3].

Author response:
In the comparison, we have included a recent SBDD baseline: TargetDiff (ICLR 2023). DecompDiff was published in a peer reviewed venue in June 2023 (ICML 2023), making it too recent (after 28.05.2024) to be expected to include in the analysis as per ICLR 2024 guidelines. DecompDiff uses a different software for estimating binding affinity (as indicated e.g. by different results for Pocket2Mol than originally reported), so it is not straightforward to compare with our method without recomputing the numbers ourselves. We will try to do that before the end of the discussion phase.

**Strong performance compared to DecompDiff.** DecompDiff takes on average 6189s to generate 100 molecules per protein pocket, making it 1.8x slower than the slowest baseline TargetDiff. Furthermore, as we argue in the paper, we believe that the ultimate goal of SBDD should be to focus on both the binding affinity and other molecular properties. DecompDiff demonstrates subpar performance in terms of most commonly used molecular properties:
* QED: FastSBDD ∈ [0.61, 0.8]; DecompDiff = 0.45
* SA: FastSBDD ∈ [0.69, 0.78]; Decompdiff = 0.61
Other metrics like LogP or Lipinski are not reported.

Reviewer comment (Evaluation):
The main question is about how much protein-ligand interation information is included in unlabeled graphs. More analyses are needed.

Author response:
Empirically, we have shown that using the unlabelled molecular graphs alone is enough to predict binding scores with accuracy high enough to outperform many SBDD baselines. This is an interesting and unexpected result, especially given that this behaviour was exhibited not only by the Vina Score, but also Gnina, a newer and reportedly, significantly more accurate docking software (that we mention in Appendix A.4).

We believe that this opens up further questions about the design of SBDD methods and development of docking software, but an extensive analysis from the structural biology point of view is not in the scope of this study.

Reviewer comment (Experiments):
The proposed disentanglement doesn't make much sense to me. I wonder whether the authors redock molecules with AutoDock Vina in their preliminary experiments (Sec 3.1 and Appendix A). I suspect that changing initial 3D conformation can also achieve a high $R^2$ of Vina score is caused by redocking. Otherwise, the Vina score should have a large change.

Author response:
*The proposed disentanglement doesn’t make much sense to me. I wonder whether the authors redock molecules with AutoDock Vina in their preliminary experiments (Sec 3.1 and Appendix A). I suspect that changing initial 3D conformation can also achieve a high $R^2$ of Vina score is caused by redocking. Otherwise, the Vina score should have a large change.*

**Role of redocking.** Yes, redocking is precisely the reason why coordinates have marginal effect on the Vina Score. The Vina Score that we and other baselines use [1, 2] for estimating docking scores does use redocking. We make it explicit in Eq (16) in Appendix A.1 where we provide a detailed definition of
the Vina Score.

Reviewer comment (Theory):
The assumption justification is needed about why the unlabelled graph sampler could be independent of protein.

Author response:
*The assumption justification is needed about why the unlabelled graph sampler
could be independent of protein*

**Justification for assumption about conditional independence.** We assume the Reviewer meant ”why the **atom sampler** could be independent of protein”, because the unlabelled graph sampler is not independent of the protein. Eq (3) explicitly shows the dependence of the unlabelled graph sampler
on the protein.

The conclusion that the **atom sampler** can be effectively taken to be independent of the protein is drawn from the experiments mentioned in Section 3.1 and detailed in Appendix A. This choice is motivated by empirical findings based on the software approximating the docking scores. Please refer to our General Response for more details.

Reviewer comment (Experiments):
The proposed method has many filtering operations (filter high-affinity molecules with binding affinity scoring model, filter high QED+SA molecules in the atom type assignment phase) during the generation process. It would be fairer to compare with optimization-based models or compare with generative models and apply a similar property screening operation as post-processing.

Author response:
*The proposed method has many filtering operations (filter high-affinity molecules with binding affinity scoring model, filter high QED+SA molecules in the atom type assignment phase) during the generation process. It would be fairer to compare with optimization-based models or compare with generative models and apply a similar property screening operation as post-processing.*

This is a great point. We view our method as bridging the gap between the generative optimization-based models. To the best of our knowledge, we are the first to introduce the explicit optimization of binding (and other molecular properties) to a generative model without having to rely on expensive computations. On the contrary, our method is lightning fast compared to baseline generative models for SBDD, which are themselves faster than optimization based models. Please see our General Response for more details.

**Post-hoc screening would result in an even more significant relative speedup.** Regarding screening predictions of generative models as post-processing, the scoring model that we use for filtering is proprietary and is part of our contribution. If we compared other baseline models and filtered their predictions using the Vina software this would come at a cost. Scoring a single protein-ligand pair takes on average 16 seconds. Assuming that we generated double the number of predictions, scored them all with Vina and filtered out bottom half, this would quadruple the runtime of the fastest baseline, which already is two orders of magnitude slower than the slowest variant of our method. It is also not obvious whether optimization for binding alone would not hurt other molecular properties. We demonstrate with our method that we can do both: outperform baselines in terms of both binding and other molecular properties.

Reviewer comment (Novelty):
The proposed method can not generate novel 2D graph structures. I'm not sure whether it's a real limitation. If the authors can justify all 2D graph structures can be covered by the ligand database, it's also fine.

Author response:
*The proposed method can not generate novel 2D graph structures. I’m not sure whether it’s a real limitation. If the authors can justify all 2D graph structures can be covered by the ligand database, it’s also fine.*

Thank you for the opportunity to clarify this. As we mention in Section 3.2, it is possible to extend the method to allow for sampling entirely new 2D graph structures, but we decide not to do that. We note that there is substantial variability in the molecular space even with a fixed set of 2D unlabelled graph structures. This is supported by the fact that our method has comparable chemical diversity to other methods as shown in Table 1.

**Diversity of the generated molecules.** To demonstrate the diversity of the molecular space with fixed 2D unlabelled graph structures, we conducted an additional experiment. We showed that we can generate hundreds (and in 98% of the cases at least a thousand) of novel and chemically diverse molecule for a fixed 2D unlabelled structure. Please see our General Response for more details. 

Regarding the novelty of generated molecules, it is true that we cannot generate unseel 2D graph structures, but the FastSBDD-$\mathcal{ND}$ and FastSBDD-$\mathcal{PO}$ variants of our model achieve 100% novelty, meaning that none of the generated molecules are present in the train set.

Reviewer comment (Evaluation):
How accurate is the binding affinity scoring model? Typically, the model utilized 3D information would be more accurate, but the proposed scoring model only uses the unlabelled 2D molecular graph.

Author response:
*How accurate is the binding affinity scoring model? Typically, the model utilized 3D information would be more accurate, but the proposed scoring model only uses the unlabelled 2D molecular graph*

**Accuracy of the binding affinity scoring model.** The binding affinity scoring model achieves 71% correlation with the ground truth on the validation set.

As we mention in section 3.1 and provide more details in Appendix A, there is marginal effect of the exact 3D configuration of the molecule on the Vina Score as it performs redocking as part of its estimation of the docking scores. Of course, the scoring model cannot be 100% accurate, because it does not use atom type information. This is done on purpose, because (1) it allows for sampling novel drugs and (2) we show empirically that this is enough to outperform
existing methods in terms of predicted binding affinity.

Reviewer comment (Theory):
In theoretical analysis, LU-3DGNN can distinguish G1, G2 with protein-ligand inter message passing. Does it mean considering ligand 3D information will lead to a better model?

Author response:
*In theoretical analysis, LU-3DGNN can distinguish G1, G2 with protein-ligand inter message passing. Does it mean considering ligand 3D information will lead to a better model?*

**Effect of incorporating 3D information on expressivity and generalization.** This is a great observation. LU-3DGNNs are indeed more expressive than LU-GNNs, because they use more information and in general can model 3D graphs better. That said, in this project, we have shown empirically that 3D information does not contain useful information for predicting the binding scores (because of redocking) and therefore our design choice to model unlabelled graphs without 3D information was motivated by empirical findings. In such scenarios, the increase in representational capacity (i.e., expressivity) using powerful models that accommodate the aditional spatial information can lead to worse generalization so the predictive performance can suffer, i.e., we may not get a better model. In fact, this may explain why approaches such as the one proposed in this work that try to balance expressivity and generalization tend to perform better than the existing approaches that focus solely on expressivity at the expense of generalization.