Paper: MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions (ICLR-2024)

Reviewer comment (Experiments):
From Table 1, it seems the proposed method gets marked improvements in both metrics across both RoseTTAFold and AlphaFold2 on CASP14 and CASP15. Do you have any ideas and insights for CASP16?

Author response:
In our study, we strategically chose CASP14 and CASP15 as they represent the latest and most renowned benchmarks in protein structure prediction. Our decision was guided by the prominence of these benchmarks in the field. To the best of our knowledge, CASP16 has not been released at the time of our research. However, we are confident in the applicability of our method to future benchmarks, as evidenced by our encouraging results on CASP14 and CASP15. It's important to note that our current model is relatively modest in scale, with limitations in both parameter size and the extent of the pretraining dataset. We anticipate significant enhancements in performance with the expansion of our model, both in terms of its complexity and the breadth of training data.

Reviewer comment (Experiments):
The proposed MSA-Generator includes Tied-Row Attention, Cross-Row Attention, and Self/Cross-Column Attention, which may have appeared in previous models like AlphaFold2. However, it is promising that the proposed model can obtain remarkable improvements on CASP14 and 15. For more ablation studies, could you provide results to illustrate which part is crucial and explain why it works for the proposed Tied-Row Attention, Cross-Row Attention, and Self/Cross-Column Attention?

Author response:
AlphaFold2 indeed utilizes an attention mechanism; however, its application differs from ours as it is not designed for generative purposes and thus does not incorporate a cross-attention mechanism. Our proposed MSA-Generator, in contrast, is specifically tailored to leverage various forms of attention to enhance generative capability.

1. **Tied-Row Attention:** This component, whose efficacy is supported by research in the MSA Transformer field, is critical in our framework. It helps in accurately capturing the evolutionary information present in MSAs, a key aspect that drives our model's performance.
2. **Cross-Row Attention:** This feature plays a vital role in reducing computational complexity. It is essential for the decoder to efficiently process the entire MSA input, enabling our model to handle large-scale data more effectively.
3. **Self-Column Attention:** Consistent with prevailing insights from MSA-related studies, we recognize self-column attention as crucial for accurately modeling MSAs. Our framework incorporates this to better understand intra-column relationships, thereby enhancing prediction accuracy.
4. **Cross-Column Attention:** We have specifically designed this to uphold the inductive bias related to the conservation property of MSAs. It adds a layer of sophistication to our model, allowing it to capture subtle but crucial evolutionary patterns.

Regarding the possibility of conducting ablation studies, it is important to note that our framework's components are deeply integrated. Removing any one of them would not only diminish the model's effectiveness but also require extensive pretraining on large datasets. Hence, conducting such studies is not feasible within our limited computation resource in this period. We do, however, acknowledge the importance of these studies and plan to undertake them in our future work. This will allow us to further validate the individual contributions of each component to the overall effectiveness of our model.

Reviewer comment (Presentation):
In fig. 4 (b), what is the meaning of the dashed lines? Why do they seem thin in the middle and thick on the sides? In fig.4 (c), why are there more improvements if intervals are less than 10 for virtual over baseline and virtual over real?

Author response:
In Fig. 4(b), created using *GraphPad Prism 9*, the solid line represents the linear regression equation, while the dashed lines indicate the 95% confidence intervals, automatically plotted by the software.

Regarding your observant question about the performance in certain cases, it's true that sequences with a baseline depth of 5 occasionally outperform those with a real depth of 15, as seen in examples like T1035-D1, T1074s1-D1, T1048-D1 (refer to Tables 5 and 6 in the Appendix). However, generally speaking, there is a proportional relationship between depth and lddt on average.

Reviewer comment (Reproducibility):
Where are the codes, do you have any plans to public code?

Author response:
Certainly, we release our codebase including inference and pretraining for future research. Here is the link to the anonymous codebase: https://anonymous.4open.science/r/MSA-Augmentor-E2D7/README.md

Reviewer comment (Novelty):
The authors' proposed task shares a fundamental similarity with the task of modeling the distribution of a given MSA. While their approach shows promise, it is crucial to provide a more comprehensive perspective and clarify the novelty of their contribution. Unfortunately, the manuscript overlooks several pertinent references, including:  
1. McGee, F., Hauri, S., Novinger, Q. et al. The generative capacity of probabilistic protein sequence models. Nat Commun 12, 6302 (2021).  
2. Riesselman, A.J., Ingraham, J.B. & Marks, D.S. Deep generative models of genetic variation capture the effects of mutations. Nat Methods 15, 816–822 (2018).  
3. Sinai, S., Kelsic, E., Church, G. M. & Nowak, M. A. Variational auto-encoding of protein sequences. NeurIPS 2017 MLCB Workshop

Author response:
We appreciate the reviewer's suggestions and will include them in our revision. Our work differs from the VAE-based methods mentioned in key ways:

1. Our method is tailored for MSA generation, emphasizing unsupervised generative pretraining for zero-shot downstream tasks. This is a capability not present in the cited VAE-based methods, which demonstrate a limited capacity in producing effective sequences **across diverse domains**.
2. Unlike VAE works focused on evaluating and simulate protein mutations and their functional implication, our focus is on novel MSA generation for improved protein structure prediction. We have included Potts model which can generate MSAs in our comparison.
3. Our main contribution is innovative pretraining tasks and a large-scale pretrained model ready to use for enhancing AlphaFold2 and others, distinct from prior research.
4. Learning and sampling from MSA distributions is more complex than it appears within VAE framework, especially with the diversity of protein families and long protein sequences. We did not see improvement on structure prediction task in relevant works. Inspired by NLP advancements, we provide an effective solution for MSA generation, directly confronting these complexities.

Reviewer comment (Experiments):
Examining Figure 4a, it is not immediately clear whether the depth or quality of the MSA holds more significance. There is uncertainty about how introducing random protein sequences into the MSA would impact performance, highlighting the need for a more in-depth analysis to understand the relationship between MSA depth and quality, and their potential trade-offs in achieving optimal results.

Author response:
The link between MSA depth, quality, and downstream performance is clear. For details, see AlphaFold2's paper "Highly accurate protein structure prediction with AlphaFold", specifically Fig 5 (a), discussing how shallower MSA depth reduces quality and impairs downstream tasks. This was our initial motivation.

Our Fig 4(a), a violin plot, demonstrates that a baseline depth of 5 generally underperforms against real and virtual depths of 15. This is evident from the blue distribution (Baseline depth=5) being more concentrated in lower scoring areas compared to the yellow (Real depth=15) and red (Virtual depth=15).

Regarding random protein sequences, our 'Iterative Unmasking' baseline already employed a similar concept, unmasking random tokens with the MSA Transformer. Yet, as Table 1 shows, this didn't significantly improve results. Addressing your concern, we tested introducing random sequences by replacing 5%, 10%, and 15% of tokens, increasing MSA depth. This worsened downstream performance, confirming our approach's effectiveness.
|  | CSAP14-plddt | CSAP14-lddt | CSAP15-plddt | CSAP15-lddt |
| --- | --- | --- | --- | --- |
| AlphaFold2 | 65.1 | 53.2 | 65.1 | 55.6 |
| AlphaFold2+5%random | 64.3 | 50.6 | 62.3 | 50.9 |
| AlphaFold2+10%random | 60.5 | 49.5 | 59.6 | 43.2 |
| AlphaFold2+15%random | 55.6 | 44.3 | 53.8 | 39.6 |
| AlphaFold2+Ours | 71.6 | 57.5 | 73.7 | 63.7 |

Reviewer comment (Experiments):
It is unclear whether masking certain sequences within the original MSA and replacing them with random protein sequences could also achieve good performance, suggesting the need for further experimentation.

Author response:
The link between MSA depth, quality, and downstream performance is clear. For details, see AlphaFold2's paper "Highly accurate protein structure prediction with AlphaFold", specifically Fig 5 (a), discussing how shallower MSA depth reduces quality and impairs downstream tasks. This was our initial motivation.

Our Fig 4(a), a violin plot, demonstrates that a baseline depth of 5 generally underperforms against real and virtual depths of 15. This is evident from the blue distribution (Baseline depth=5) being more concentrated in lower scoring areas compared to the yellow (Real depth=15) and red (Virtual depth=15).

Regarding random protein sequences, our 'Iterative Unmasking' baseline already employed a similar concept, unmasking random tokens with the MSA Transformer. Yet, as Table 1 shows, this didn't significantly improve results. Addressing your concern, we tested introducing random sequences by replacing 5%, 10%, and 15% of tokens, increasing MSA depth. This worsened downstream performance, confirming our approach's effectiveness.
|  | CSAP14-plddt | CSAP14-lddt | CSAP15-plddt | CSAP15-lddt |
| --- | --- | --- | --- | --- |
| AlphaFold2 | 65.1 | 53.2 | 65.1 | 55.6 |
| AlphaFold2+5%random | 64.3 | 50.6 | 62.3 | 50.9 |
| AlphaFold2+10%random | 60.5 | 49.5 | 59.6 | 43.2 |
| AlphaFold2+15%random | 55.6 | 44.3 | 53.8 | 39.6 |
| AlphaFold2+Ours | 71.6 | 57.5 | 73.7 | 63.7 |

Reviewer comment (Evaluation):
In Figure 7, there is a lack of clarity about the methodology behind calculating the conservation and quality metrics. It appears that these scores are derived solely from the provided sequences, such as the six sequences for T1093-D1, which raises questions about the significance of these metrics.

Author response:
We would like to clarify that Figure 7 in the Appendix serves **purely for visualization purposes** of the MSAs generated and is **not a basis for analytical evaluation in our study**. For visualization, we use Jalview, a standard tool that automatically calculates metrics like quality and consensus. However, **these metrics are not the focus of our experiments in the main content of the paper**. They are relevant only for single MSA analysis and **do not provide an accurate assessment of our methods**. We have **not employed these metrics in any part of our research presented** in the paper. The Diversity and Conservation metrics we use are derived from PSSM and Shannon Entropy, as detailed in section 4.5, and Figure 7 is not connected to this analysis.

These metrics are calculated per column and don't indicate overall MSA quality and conservation. Our model doesn't enforce conservation but learns it via cross-column attention. Fig 7's metrics don't show overall MSA quality, and we didn't target them specifically. We've removed this unclear part in our revision.

In our prior response, we detailed outcomes for sequences generated via random replacement; see those for details.

Reviewer comment (Evaluation):
In Section 4.5, the authors compare their method with Iterative Unmask but do not provide a comparison of the pLDDT scores using this baseline method.

Author response:
Section 4.5 showcases the diversity of generated MSAs, **focusing on analyzing these MSAs directly** instead of downstream tasks like protein structure prediction. Thus, we omitted a comparison of pLDDT scores here, but it's available in section 4.3 and table 1.

To better clarify:

- We describe that '*The results, derived from 10 runs following section 4.3*' showing dataset and strategies are consistent with Section 4.3.
- The construction of the Position-Specific Scoring Matrix (PSSM) is detailed in the middle part of Section 4.5 as “*we examined the PSSM of both **original** and **MSA-Generator generated MSAs***”
- To measure the similarity between generated and original MSAs' PSSMs, we used the **Pearson correlation coefficient**. Below is the pseudocode for this computation:

```python
gen_pssm = calculate_pssm(gen_msas_seqs[i], bg_freqs)
source_pssm = calculate_pssm(source_msas_seqs[i], bg_freqs)
correlation = np.corrcoef(source_pssm.ravel(), gen_pssm.ravel())[0, 1]
```

we have revised our draft for clear presentation.

Reviewer comment (Experiments):
There is ambiguity regarding the dataset used for the analysis in Section 4.5. Specifically, it is unclear which sets of sequences were used to construct the Position-Specific Scoring Matrix (PSSM) and what the correlation coefficients are when employing Iterative Unmask.

Author response:
Section 4.5 showcases the diversity of generated MSAs, **focusing on analyzing these MSAs directly** instead of downstream tasks like protein structure prediction. Thus, we omitted a comparison of pLDDT scores here, but it's available in section 4.3 and table 1.

To better clarify:

- We describe that '*The results, derived from 10 runs following section 4.3*' showing dataset and strategies are consistent with Section 4.3.
- The construction of the Position-Specific Scoring Matrix (PSSM) is detailed in the middle part of Section 4.5 as “*we examined the PSSM of both **original** and **MSA-Generator generated MSAs***”
- To measure the similarity between generated and original MSAs' PSSMs, we used the **Pearson correlation coefficient**. Below is the pseudocode for this computation:

```python
gen_pssm = calculate_pssm(gen_msas_seqs[i], bg_freqs)
source_pssm = calculate_pssm(source_msas_seqs[i], bg_freqs)
correlation = np.corrcoef(source_pssm.ravel(), gen_pssm.ravel())[0, 1]
```

we have revised our draft for clear presentation.

Reviewer comment (Theory):
The underlying rationale for the superiority of the MSAGenerator over traditional MSA search methods remains ambiguous, especially given that the pretraining of the model is based on the outputs from MSA searches. This warrants further elucidation.

Author response:
To clarify, our approach is **not intended to replace actual multiple sequence alignments (MSAs) with synthetic alternatives**. As we highlighted in the introduction of our manuscript, our main focus is on proteins characterized by a limited number of homologous sequences. In these instances, traditional MSA search algorithms often fail to construct comprehensive alignments. Our methodology, therefore, arises out of necessity: **we generate the required MSAs when existing data is insufficient**. An illustrative example is the protein T1122 from CASP15, identified in the Tipula oleracea nudivirus. The scarcity of homologous sequences for this protein in existing databases presents a significant obstacle in constructing MSAs, which are vital for the analysis of protein structure and function.

Our MSAGenerator is designed to specifically address this gap, generating virtual sequences to effectively supplement real sequences, especially in scenarios where actual sequence acquisition is challenging or not feasible. Notably, our model, pretrained on a diverse array of searched sequences, is adept at learning complex co-evolutionary patterns within MSAs. This learning approach enables it to generalize to a broader domain, similar to the capabilities of a Large Language Model. This functionality significantly enhances downstream structural predictions, a claim substantiated by the experimental results we present.

Reviewer comment (Experiments):
There is a disparity in the choice of databases: the paper mentions using the UniRef50 database for pretraining and then switches to UniRef90 for testing. The reasons for this choice are not explained, and it raises questions about the potential variation in results had UniRef50 been employed for both stages.

Author response:
Thank you for pointing out this typo. We apologize for any confusion caused by the apparent inconsistency in our manuscript. To clarify, we consistently used the UniRef90 database for both the pretraining and testing phases of our study, UniRef50 was not employed throughout our study

Reviewer comment (Experiments):
The introduction of an "artificial challenging MSA" requires a more comprehensive explanation. Its significance and the rationale behind its inclusion in the study are unclear. Figure 4a seems to depict an imbalanced comparison. If homologous sequences were present during the pretraining phase but not considered for the baseline, it would be an unfair representation. This suggests that the MSAGenerator's performance may not surpass real MSAs.

Author response:
The concept of the *artificial challenging MSA* is introduced in our study to critically assess the efficacy of our generated MSAs in comparison with real MSAs. We wish to clarify that our intention is not to **suggest the exclusive use of virtual MSAs over real ones, nor do we claim that virtual MSAs are inherently superior** (please refer to response 1)**.** Our primary objective is to demonstrate that virtual MSAs can closely approximate real ones in downstream tasks.

The inclusion of this comparison is vital to establish the utility of our MSAGenerator in situations where real MSAs cannot be obtained, as discussed in section 4.3. Before proposing the use of virtual MSAs in such contexts, it is essential to validate their effectiveness in comparison to real MSAs derived from traditional search methods. We have renamed this section into “Are Virtual MSAs as good as Real MSAs?” for better presentation.

We also want to emphasize that none of the testing sequences were included in any part of our pretraining data. This ensures a fair and unbiased assessment of the MSAGenerator’s performance

Reviewer comment (Presentation):
For Figure 4c, it is crucial to also represent the "Real over Virtual" comparison to provide a comprehensive view (negative interval).

Author response:
Thank you for your advice. We have incorporated the suggestion of providing a negative interval, and this has been included in the revised draft of our manuscript

Reviewer comment (Evaluation):
Questions arise regarding the consistency of the reported improvements in Figure 4 when other protein structure prediction methods, like RoseTTAFold, are used.

Author response:
We appreciate your inquiry regarding the consistency of our reported improvements when using different protein structure prediction methods, such as RoseTTAFold. In Section 4.2 of our paper, our primary goal is to demonstrate the quality of the MSAs generated by our method. This section is not intended to suggest the replacement of real MSAs with virtual MSAs when real ones are available. We chose to focus on AlphaFold2 (AF2) for these experiments because it generally outperforms RoseTTAFold.

However, recognizing the validity of your concern, we have expanded our analysis in Section 4.3, the core results section of our paper. Here, we conduct experiments using both RoseTTAFold and AF2. This section specifically addresses scenarios where no homologues are identifiable using traditional search algorithms. By including both methods in our analysis, we aim to provide a more comprehensive validation of the effectiveness of virtual MSAs in these challenging situations

Reviewer comment (Reproducibility):
In Table 1, the specific MSA methodology employed in the benchmarking process, as well as the precise parameters and databases used for MSA searches, are not detailed. It remains unclear if these are consistent with the pretraining stage.

Author response:
Thank you for your suggestion to clarify the experimental setup detailed in Table 1. In these experiments, we employed *real-world challenging MSAs* as introduced in Section 4.1. For MSA construction, we utilized JackHMMER with its default search parameters, specifically targeting the UniRef90 database. Each MSA created in this process contained fewer than 20 homologues, and these MSAs were consistently used across all experiments in this section

To elaborate further:

- For single-sequence-based algorithms like ESMFold and OmegaFold, we directly input the query sequence, bypassing the need for an MSA.
- For MSA-based algorithms, we employed various methods (Potts model, Iterative Unmasking and Ours) to generate MSAs, which were then fed into the folding algorithms.

We ensured a consistent experimental setup across all results reported in Table 1, aligning the benchmarking process closely with the conditions of our model's pretraining.