Paper: Harmonic Prior Flow Matching for Multi-Ligand Docking and Binding Site Design (ICLR-2024)

Reviewer comment (Evaluation):
Is there any post hoc method to generate side-chains for the designed binding site? If so, from my perspective, evaluation on binding affinity of generated binding sites and given ligands is more rationale.

Author response:
### Questions:
**“Is there any post hoc method to generated side-chains for the designed binding site?”**

A great question! Multiple methods exist for the task of sidechain packing; however, they commonly are not aware of the ligand. In the updated paper, we improved our discussion in section 4.3 on binding affinity evaluation with respect to how, e.g., multiple residue types might be able to bind a particular ligand. We attempted to address this by developing “BLOSUM score,” which takes evolutionary and physicochemical similarity into account. Ultimately, we think a final evaluation has to be biological validation.

Reviewer comment (Novelty):
The problem of multi-ligand binding is interesting, but I am not sure how important this task is. The authors briefly discussed this in Sec 2, and more detailed explanations can be helpful.

Author response:
**1, First bullet: “The problem of multi-ligand binding is interesting, but I am not sure how important this task is [...]  more detailed explanations can be helpful”**

* Multi-ligand binding is indeed an important problem since it is crucial for, e.g., fragment-based drug design, and in our context, it is of interest for enzyme design where we would desire to design a binding site for multiple reactants / starting molecules of a chemical reaction that the enzyme should catalyze.

Reviewer comment (Writing):
Does this vector field $u$ in Eq 1 have any physical meaning?

Author response:
**3, Third bullet: “Does this vector field u in Eq 1 have any physical meaning?”**

* An excellent question! In general we want to be careful with such physics interpretations. The direction the vector field points in is where the expected x_1 is given that we currently are at x_t. At times very close to t=1 this can become similar to the direction of the forces on the atoms if our training data is from the Boltzmann distribution of the molecules.

Reviewer comment (Theory):
Can authors help summarize the essential difference between score matching / denoising diffusion with the objective (Eq 1&3) in this paper? Because score matching and denoising diffusion can be treated as ODE flow models. Also, is the proposed HarmonicFlow specific for multi-ligand docking? Now it seems that it can also fit for the general protein-ligand docking.

Author response:
**4.1 Fourth bullet first part: “summarize the essential difference between score matching / denoising diffusion with the objective (Eq 1&3) in this paper?”**

* Another great question in our opinion! Flow matching and denoising diffusion are indeed similar, with flow matching being a generalization of diffusion models. The quantity $u_t(x | x_0, x_1)$ that we regress against in equation 3 is a conditional vector field that points to a data sample (obtained by, e.g., the vector $x_1 - x_0$). Meanwhile, in diffusion models, we regress against the conditional score, which is a vector pointing toward the steepest ascent in a noised distribution. This way, in flow matching, we learn a drift that we can integrate to sample while in diffusion models, we have a predefined arbitrary drift that we have to combine with the learned score to sample. Please let us know if we can clarify this any further and if any questions remain!

**4.2 Fourth bullet second part: “is the proposed HarmonicFlow specific for multi-ligand docking?”**

* HarmonicFlow is not specific to multi-ligand docking and can also be used for single-ligand docking which we hope is now better made clear in the updated text!

Reviewer comment (Experiments):
Many baselines, such as the related works mentioned in Sec 2, are missing in the experiments.

Author response:
**5, Fifth bullet: “Many baselines, such as the related works mentioned in Sec 2, are missing in the experiments.”**

* We adapt our related work section and baseline explanation in the experiments to better reflect the following:

* With respect to HarmonicFlow: the works in section 2 (apart from DiffDock’s diffusion process, which we compare with) all require sidechain positions of the protein as input which is not available in our final application of interest — binding site design — where we aim to predict the side-chain identities.

* With respect to FlowSite: of the mentioned inverse folding methods, we chose PiFold since it currently has the state-of-the-art recovery rate.

Reviewer comment (Novelty):
Most of the technology is just an application of the existing methods which prevents the paper from getting higher scores in machine learning conferences.

Author response:
We think our work provides 3 main important pieces of novelty: 1. We provide the first flow matching process to jointly generate discrete and continuous data, which we consider a novelty given that there is no work on discrete flow matching yet. 2. We think next to technical novelty, the novelty of the application is also important - our work is the first to tackle binding site design with deep learning. 3. Ours is (among another ICLR submission) the first to investigate flow matching for biomolecular structures.

Reviewer comment (Evaluation):
How fast are the two methods in terms of inference speed?

Author response:
An excellent question and we provide these details in the updated manuscript! Averaged over 4350 generated samples, the average runtime of HarmonicFlow is 0.223 seconds per generated structure and that of FlowSite is 0.351 seconds per sequence.

Reviewer comment (Novelty):
Overall the paper is much of a summarization of known techs: harmonic priors are discussed in EigenFold; self-conditioning is applied in CV/ RFdiffusion; replacing diffusions with flow matching is well studied in CV; and model architecture improvements are somehow trivial.

Author response:
* We argue that there are 3 critical points of novelty that might be overlooked. We provide the first flow matching process to jointly generate discrete and continuous data, which we consider a novelty given that there is no work on discrete flow matching yet. Secondly, we think next to technical novelty, the novelty of the application is also important - our work is the first to tackle binding site design with deep learning. Thirdly, ours is (among another ICLR submission) the first to investigate flow matching for biomolecular structures, which exhibit (also in diffusion models) different behavior than CV.

Reviewer comment (Experiments):
Three applications are proposed in this paper, while except for docking, multiple docking / binding site generation are not well investigated. The idea of multiple ligand docking itself is interesting, but not well studied. As it is mentioned in the title I'd expect some impressive case studies of multiple docking and some analysis of significance in the result. I would suggest the authors be more focused.

Author response:
* Thank you for the suggestion! We now included several case studies that were randomly picked from the test set and show how HarmonicFlow produces more realistic bond lengths and bond angles and respects physical constraints better. We acknowledge that our work indeed does not focus on multi ligand docking - we describe the problem and evaluate it since we see it as important for binding site design when the goal is to design binders for multiple molecules, such as reactants in an enzyme. In our updated version, we clarify this focus - we are happy to also change the title if you think that is a better description of the work.

Reviewer comment (Experiments):
Baselines are weak.

Author response:
* We added another new stronger baseline to Table 3 based on the anonymous ICLR 2024 submission DiffDock-Pocket [1]. It operates similar to our ground truth position comparison model, except that the fixed positions are given by a sample from DiffDock-Pocket, a recent state-of-the-art pocket level docking tool. We note that, as we are the first to consider this task, there are no existing baselines for it yet.

| Method              | BLOSUM Score | Recovery Percentage |
|---------------------|--------------|---------------------|
| DiffDock-Pocket Pos | 42.6         | 45.0                |
| FlowSite (ours)     | 47.6         | 49.5                |

Reviewer comment (Experiments):
Are sidechains configurations by any means included as data inputs?

Author response:
* They are not used as inputs and only a quantity that we predict as auxiliary target.

Reviewer comment (Theory):
I would expect some analysis of the ODE dynamics, especially with regard to the joint dynamics of residue types and molecular conformations.

Author response:
* Thank you for the suggestion! We now included a plot (Figure 11) of the evolution of FlowSite's x_1 prediction's RMSD and the evolution of the output entropy of the residue type probabilities. The results show how a more determined structure prediction correlates with a decrease in uncertainty of the residue type prediction.

Reviewer comment (Presentation):
The paper is rather hard to follow.

Author response:
We understand how these questions arose - we updated the text to contain the following answers to your questions. We hope this makes the paper easy to follow and the experiment setup sufficiently clear. Please let us know if further changes are required.

Reviewer comment (Experiments):
The design of most experiments is not clear to me (see questions).

Author response:
We understand how these questions arose - we updated the text to contain the following answers to your questions. We hope this makes the paper easy to follow and the experiment setup sufficiently clear. Please let us know if further changes are required.

Reviewer comment (Evaluation):
In Q2, you compare the ground truth amino acids with the predicted ones. Is it a fair surrogate metric for de novo generation, when for a given set of ligands and backbone coordinates there potentially can be many different answers? An algorithm that can generate pockets with a better binding affinity than affinity of pockets from the dataset, will have a low score.

Author response:
2, **Regarding using amino acid recovery for known pockets as metrics and that there can be multiple right answers, not only the residues in the ground truth data.**

* This is correct and an important point! The recovery metric has been shown to be a good proxy for downstream performance in the inverse folding literature (https://www.science.org/doi/10.1126/science.add2187 PMPNN, https://arxiv.org/abs/2306.16819 GRADEIF) and we try to improve over it with our BLOSUM metric which allows for multiple solutions by taking evolutionary and physicochemical similarity of residues into account. We now better discuss your concern in the 4.3 Metrics paragraph, and in our conclusion, we acknowledge how, ultimately, biological validation is required to fully evaluate, which is ongoing work.

Reviewer comment (Experiments):
The output of the flow-matching model should also depend on the sampled initial conditions. In Q2 experiments, do you consider a single run of the noise sampling? Have you tried to sample many different noise vectors and compare the outputs?

Author response:
3, **Question “Have you tried to sample many different noise vectors and compare the outputs?”**

* We indeed generate 10 samples per complex and evaluate the average quality. In the updated text, we now also provide the standard deviation over these samples, which is 1.16 Angstrom RMSD for Distance-Pockets on the time split (similar for all settings) and 
2.5% for the recovery rate and 1.8 Blosum score on PDBBind.

Reviewer comment (Evaluation):
You said that it is not feasible to estimate the energy. Why? I suppose that the energy of the designed protein and the affinity of the complexes can be somehow estimated using traditional or machine learning methods. At least, it is possible to use the all-atoms confidence model from DiffDock and compare scores of complexes.

Author response:
4, **Estimating binding free energy or using DiffDock’s confidence model as proxy.**

* Our discussion of the following was unclear in the text, and we improved it in the update: DiffDock’s confidence model and common computational binding affinity correlates would require the ligand and side chain atom positions as input, which none of the methods (except ours) produce. Thus, we could only produce a number for FlowSite with no comparison. Further, DiffDock’s confidence model was never trained on negatives, meaning that it cannot distinguish between binders and non-binders, and there is no evidence for it as a correlate for binding affinity.

Reviewer comment (Experiments):
In many experiments, you mentioned the oracle method that has access to the ground truth ligand structure. What exactly is this method?

Author response:
5, **“you mentioned the oracle method that has access to the ground truth ligand structure. What exactly is this method?”**

* Thank you for pointing out our insufficient explanation, which we now improved! This is the same architecture as FlowSite, but instead of using the predicted ligand positions, the ground truth crystal structure ligand positions are used as input and not altered. In practice, this information would not be known during inference.

Reviewer comment (Reproducibility):
How much time does it take to train your model on PDBBind and MOAD?

Author response:
6, **Question: “How much time does it take to train your model on PDBBind and MOAD?”**

* We now included this in the paper: on PDBBind, FlowSite took 58.6 hours, and on MOAD 115.6 hours. Both on an RTX A600 GPU.