Paper: Bridging Sequence and Structure: Latent Diffusion for Conditional Protein Generation (ICLR-2024)

Reviewer comment (Experiments):
The paper mentions that the method supports "protein-protein docking with contact information" tasks, which may not be very practical, as in real-world applications, blind docking is more common.

Author response:
[_Docking with contact information may not be very practical._] We emphasize that our approach also readily performs blind docking, as explained in section 4 in the paper, and evaluated in the experiments section. While contact information may not always be available in practical applications, there are experimental cases where it may be (e.g. a given pocket is known). We believe that studying how the method can use this additional information and how it affects its performance to be an interesting direction, and that’s why we include it in addition to blind docking.

Reviewer comment (Novelty):
The method's innovativeness is limited. This method essentially involves multi-modal data input and the use of latent diffusion.

Author response:
[_Limited innovativeness._] While we respect the reviewer's perspective, we respectfully disagree and remain confident that our paper offers valuable insights and innovative contributions to the field. The unique combination of the components of OmniProt and the seamless application to diverse biological tasks like protein-protein docking and inverse folding is novel and innovative. This is aligned with a lot of work in this domain, which also applies diffusion models (over euclidean spaces or manifolds) for different protein generation tasks. We believe our work has the potential to shift the field by demonstrating the versatility of latent diffusion in addressing complex challenges in structural biology.

Reviewer comment (Evaluation):
For the docking task, for methods based on generative models, such as the method proposed in this paper and the compared baseline DiffDock, the paper reports "oracle" statistics by selecting the prediction with the lowest RMSD from the ground truth among the 20 generated results. I believe it would be valuable to also present the mean and variance of these 20 samples on each metric. In real-world applications, we do not have ground-truth complex structures as a reference.

Author response:
[_Docking benchmark use of oracle metrics, mean and variance._] Using oracle statistics to compare generative models is common practice (see, for instance, DiffDock) as it offers insights into the upper limit of a method’s prediction accuracy, particularly when comparing to other baselines, ensuring a fair assessment of each method. For real-world generative modeling processes, it is common to introduce discriminative oracles as a substitute for known true conformations.

Reviewer comment (Evaluation):
Although the proposed model is claimed to "bridge sequence and structure," Table 3 shows that the proposed method underperforms the inverse folding baseline on all metrics. This raises questions about the effectiveness and motivation of the unified framework by modeling both seq-structure as claimed by the authors. In the docking evaluation, OMNIPROT lags behind the baseline DockGPT by a significant margin for high-quality metrics (25th quantile for RMSD and ≥high for DockQ). This suggests that OMNIPROT struggles to recall the most accurate "hits" but generates many just "acceptable" docked poses. In traditional docking, candidate poses are searched and ranked by free energy, with only a few top (low energy) poses selected for downstream analysis or tasks. Therefore, the high-quality metrics for docking should be given the utmost consideration during evaluation, as the results weaken OMNIPROT's performance. The authors are welcome to argue against this if they disagree.

Author response:
[_Metrics, inverse folding and docking._] While we acknowledge the observed underperformance of our proposed model on some metrics, we believe it is crucial to highlight the specific design philosophy behind OmniProt, as a single model to tackle multiple tasks in the protein generation space. Unlike methods focused on a single "hit", OmniProt is designed as a generative model with the intent of providing a diverse set of poses. The emphasis here is on capturing the broad spectrum of feasible solutions that could be relevant in certain biological contexts rather than solely optimizing for the most accurate single pose. As can be seen in the results, it produces more acceptable and medium poses, which means that it is able to find good poses for cases where DockGPT fails. That being said, we believe OmniProt metrics may further improve once we train it with the updated autoencoder. Also, it is worth emphasizing that OmniProt significantly outperforms the previous state of the art diffusion for docking (DiffDock-PP).

Reviewer comment (Experiments):
The paper lacks an explanation for the design choices made for the OMNIPROT architecture. There is also no ablation study on the proposed framework, encompassing the encoder, decoder, and denoising network. Additionally, the authors should clarify why they adopted latent diffusion instead of a vanilla VAE or other sampling strategies used in VAE, providing corresponding evidence or support, such as an ablation study.

Author response:
[_Ablation study for architecture._] The combination of individual components in OmniProt is unique, yet, the choices for both the autoencoder and diffusion model are fairly standard and extensive ablation studies are costly. Please see the general response for a more detailed discussion.

Reviewer comment (Experiments):
While Section 5 mentions that the training data were collected "without any quality-based filtering," the authors should clarify whether they performed redundancy removal for the newly curated docking dataset. Since the docking task leverages both sequence and structure features, proper filtering for the training set (it is mentioned in the paper as "consists of the remaining data") based on sequence (and structure) similarity is required to prevent the potential data leakage.

Author response:
[_Redundancy removal._] Thank you for pointing this out. By using the Foldseek score for clustering, which linearly combines both 3D-based structure and sequence substitution scores (Appendix F), our approach combines sequence and structure-based similarity metrics and restricts them specifically to interfaces, to provide a robust approach for a protein-protein-interaction benchmark dataset creation. As the test clusters are separate from the train clusters, the train dataset is inherently deleaked against the test dataset.

Reviewer comment (Novelty):
A related work, PROTSEED [1] (I found it already published in ICLR 2023, thus cannot be counted as concurrent work), also employs the IPA module of AF2 to decode both sequence and structures for multiple tasks, including inverse folding. The authors should clarify how OMNIPROT differs from PROTSEED and provide a better context for the reference. Failing to show significant improvement over the related works can weaken the contribution of the proposed model.

Author response:
[_Differences w.r.t. PROTSEED._] Thank you for pointing this out, we will add this to related work. While PROTSEED is also jointly encoding sequence and structure with conditioning it does so without employing diffusion. The formulation behind OmniProt uses diffusion models which were originally designed to sample distributions; in the context of protein design to sample ensembles of protein conformations. The use of diffusion leads to a different overall formulation in terms of training objective and sampling schemes. However, we do believe that PROTSEED provides a relevant baseline and we are working on adding it to our empirical evaluation.

Reviewer comment (Evaluation):
Regarding the inverse folding evaluation, what is the rationale for using sc-RMSD instead of perplexity, where the latter is more commonly evaluated for inverse folding task? The sc-RMSD/TM are typically used for backbone (structure) generation models like FrameDiff [5]. Since the authors claim the proposed model to be "generative," the perplexity (ppl) on the test set is encouraged to be reported. The ppl can be more important in reflecting the performance of the inverse folding model compared to the flawed recovery metric.

Author response:
[_IF perplexity metric._] While more metrics may always yield valuable insights, we believe perplexity does not give any indication whether a protein sequence will fold into the desired structure, which is why we chose to report sc-RMSD, an in-silico proxy for designability.

Reviewer comment (Reproducibility):
It is unclear to me whether the mentioned two tasks in this paper involve independent model training or if some (or all) of the model parameters are shared across tasks. If any parameters are shared, the authors could specify which ones. (please point to the related text if already mentioned)

Author response:
[_Model shared or re-trained for different tasks._] The autoencoder is trained once for each dataset, and corresponding diffusion models are trained for each autoencoder and dataset.

Reviewer comment (Writing):
What is the "geometrically-structured" latent space compared to the canonical latent space as mentioned in the abstract and introduction sections?

Author response:
[_Geometrically structured latent space vs. canonical._] The abstract and intro both refer to the geometrically structured latent space. We use “geometrically structured” because the representations are invariant to rigid body transformations of the input. As we explain in 4.4.1, this property means that OmniProt’s performance on conditional generative tasks is invariant to rigid body transformations of the input structures.

Reviewer comment (Writing):
Section 3 deserves a further refinement.

Author response:
[_Refining section 3 (related work)._] We are happy to expand or modify the related work section (section 3). We will add some missing papers raised in the reviews, any other comments and suggestions are welcome.

Reviewer comment (Writing):
I found the paragraph in section H (appendix) incomplete.

Author response:
[_Incomplete paragraph in Section H._] Thanks for bringing this up. We apologize for this oversight, we will fix the incomplete paragraph.

Reviewer comment (Theory):
It is unclear if the autoencoder approach introduces a fundamental limitation to the method, or if there are straightforward paths to improving the decoder.

Author response:
[_Is the autoencoder a fundamental limitation? Ablation of architecture._] No, we do not believe it is. We are currently exploring changes to the autoencoder, and were able to improve its performance significantly. Please see the general answer for further details about updated performance. Since OmniProt was working at the limit of the autoencoder performance, we believe this will improve results (e.g. high-quality metrics for protein-protein docking). The main changes to the autoencoder were in the decoder, where we added pair feature updates and increased the number of layers.

Reviewer comment (Experiments):
The authors show one example of ablating the effects of joint training on sequence and structure, showing improvement in inverse folding metrics when training with structure-based features; their approach affords unique insight into the value of joint training and this point could be expanded upon with further experiments.

Author response:
[_Experiments for further insights regarding joint seq+structure training._] We explored this for inverse folding. However, for docking all methods rely on the respective sequences as input, so it is not obvious to us how to decouple them. We agree studying this further is quite interesting, and appreciate any suggestions to do so.

Reviewer comment (Experiments):
The authors repeatedly mention that improvements to the autoencoder might improve overall OmniProt performance. Do the authors believe that this is a scaling / expressivity problem, or architectural?

Author response:
[_Is the autoencoder a fundamental limitation? Ablation of architecture._] No, we do not believe it is. We are currently exploring changes to the autoencoder, and were able to improve its performance significantly. Please see the general answer for further details about updated performance. Since OmniProt was working at the limit of the autoencoder performance, we believe this will improve results (e.g. high-quality metrics for protein-protein docking). The main changes to the autoencoder were in the decoder, where we added pair feature updates and increased the number of layers.

Reviewer comment (Evaluation):
The authors highlight that OmniProt inverse folding samples are generated using reverse diffusion, rather than the randomized autoregressive scheme used in ProteinMPNN. This does intuitively seem like a major advantage, but no metrics are reported to support this claim. Can the authors provide some evidence beyond sequence recovery that more directly relates to sample quality?

Author response:
[_Metrics beyond sequence recovery._] We believe the main advantage of OmniProt comes from the flexibility in controlling the number of steps when generating samples. For diffusion models the number of steps is independent of the protein length, while autoregressive methods require one step per residue in the protein.