Paper: RDesign: Hierarchical Data-efficient Representation Learning for Tertiary Structure-based RNA Design (ICLR-2024)

Reviewer comment (Novelty):
I find the article lacks novelty; the core multi-level representation learning relies on self-supervised methods from computer vision, which is not novel.

Author response:
**Q5** ***I find the article lacks novelty; the core multi-level representation learning relies on self-supervised methods from computer vision, which is not novel.***

**A5** **We would like to know if this is the main reason for your vote for rejection.**

We believe that our primary contributions are: (i) formulating the important biological problem of tertiary structure-based RNA design as one that can be solved by artificial intelligence techniques; (ii) preparing the data for solving this problem by carefully selecting high-quality data and proposing a rigorous partitioning based on structural similarity and sequence length distribution; (iii) addressing the issue of scarce data for this problem by proposing a data-efficient learning framework. **We cannot believe that the use of self-supervised learning methods from computer vision in our data-efficient learning framework is a reasonable reason for rejection.**

By the way, we are curious to know if you would consider increasing the score, or what we would need to do for you to consider raising it.

Reviewer comment (Experiments):
The baseline comparisons are incomplete. While the article compares some protein inverse folding methods, it fails to include recent protein inverse folding methods for comparison, such as ProteinMPNN and PiFold, which are mentioned in the related work section.

Author response:
**Q3** *The baseline comparisons are incomplete. While the article compares some protein inverse folding methods, it fails to include recent protein inverse folding methods for comparison, such as ProteinMPNN and PiFold, which are mentioned in the related work section.*

**A3** Thank you for your suggestion. In this paper, we have selected several classic methods from protein design. From an methodology perspective, ProteinMPNN and StructGNN models are fundamentally similar. However, PiFold introduces a novel approach, utilizing NodeMLP and EdgeMLP, along with Global Context Attention. We have included additional results from PiFold in the revised manuscript. Our findings indicate that while PiFold is effective in protein design, its strong model tends to overfit when applied to data-scarce RNA datasets, resulting in less impressive performance.

Reviewer comment (Reproducibility):
The dataset used in the experiments is not publicly available, making it impossible to directly compare the results with those in previously published works.

Author response:
**Q2** *Dataset Availability*

**A2** We have released the data to the following anonymous link: https://we.tl/t-eU6y2NZLLs. You can download and review it by using the cPickle package in Python.

Reviewer comment (Novelty):
The author mentions collecting data to create a dataset, which is divided by sequence length. However, this approach is widely used in the protein domain and lacks innovation. Additionally, the author does not cite articles from the protein domain as references here.

Author response:
**Q1** *Benchmark Dataset*

**A1** Our benchmark dataset primarily focuses on two core issues: (i) the scarcity of high-quality data; and (ii) the lack of reasonable split. We have curated a high-quality dataset by manually checking all available data from PDB and RNAsolo. Additionally, we ensured that the distributions of sequence lengths are essentially consistent across the train, validation, and test sets, while simultaneously ensuring that there is no data leakage between them. 

**This data partitioning method employs both structural similarity and sequence length distribution, , which we believe differs from the traditional approach of simple partitioning based on sequence length. The specific partitioning process and the analysis of the dataset are detailed in Appendix A and Appendix H, respectively.** If you find similar works that has proposed this partitioning approach, we would be pleased to add a citation.

Reviewer comment (Experiments):
I consider this to be the most critical and crucial analysis that is missing. Although the paper does compare the proposed architecture (with its multi-level contrastive losses and secondary-structure loss) to other architectures like LSTMs or graph transformers, there aren’t any comparisons to any other methods which directly tackle the RNA inverse-folding problem. There are _many_ such methods, including several that are as recent as this year (e.g. MCTS-RNA, LEARNA, aRNAque, eM2dRNAs, etc.). Not having these comparisons really holds this paper back, because there is no understanding of how this method (which is being presented as a RNA inverse-folding method) compares to any other RNA inverse-folding method.

Author response:
**A1** Thank you for your comment. In this paper, we mainly focus on the tertiary structure-based RNA design. We compare several secondary structure-based RNA design approaches that have public web servers, including RNA Designer, RNAInverse, and INFORNA. The results are shown in the table below. We have added results from secondary structure-based baseline studies to the **Appendix I** of the revised manuscript. If necessary, we will include a discussion about other secondary structure-based approaches.

| Method       | Rfam Recovery (%) | RNA-Puzzles Recovery (%)  | Rfam Macro F1 (×100) | RNA-Puzzles Macro F1 (×100)  |
|--------------|---------------|--------------------------|--------------------|----------------------------|
| SeqRNN (h=128) | 27.99±1.21       | 28.99±1.16              | 15.79±1.61           | 16.06±2.02                |
| SeqRNN (h=256) | 30.94±0.41       | 31.25±0.72              | 13.07±1.57           | 13.24±1.25                |
| SeqLSTM (h=128)| 24.96±0.46       | 25.78±0.43              | 10.13±1.24           | 10.39±1.50                |
| SeqLSTM (h=256)| 31.45±0.01       | 31.62±0.20              | 11.76±0.08           | 12.22±0.21                |
| StructMLP     | 24.40±1.63       | 24.22±1.28              | 16.79±4.01           | 16.40±3.28                |
| StructGNN     | 27.64±3.31       | 27.96±3.08              | 24.35±3.45           | 22.76±3.19                |
| GraphTrans    | 23.81±2.57       | 22.21±2.98              | 17.32±5.28           | 17.04±5.36                |
| **RNAsoft** [1]    | 26.60±5.71       | 28.44±7.31              | 25.86±5.40           | 28.01±7.51                |
| **RNAInverse** [2]    | 25.46±5.50       | 25.69±6.75              | 24.75±5.50           | 25.01±6.48                |
| **INFORNA** [3-4]       | 30.45±6.01       | 33.79±9.08              | 27.66±5.62           | 29.94±7.41                |
| **RDesign**       | **58.25±0.92**   | **50.57±0.81**          | **55.03±0.93**       | **47.06±0.60**            |

Reviewer comment (Evaluation):
Although there is some justification provided for the performance metrics focused on in the paper (i.e. recovery and Macro-F1), these metrics are still somewhat limited. Recovery is simply “accuracy” here, and although this is a great metric to report, some bases will naturally be harder than others to predict. For example, bases participating in certain secondary structures, or those that interact with other residues in the 3D structure, may be easier or harder to classify. Thus, instead of reporting recovery/accuracy pooled over the entire macromolecule, it would be useful to see these numbers stratified by different secondary structures or tertiary attributes.

Author response:
**A2** Thank you for your insightful comment. Our work primarily focuses on RNA design based on tertiary structure, whereas the performance metrics are largely referenced in protein design. We found that the recovery metric is an intuitive one, while the commonly-used perplexity metric is limited to this work.

Your suggestion is valuable. We have added such evaluation in the **Appendix J**. It can be seen that our approach consistently outperforms the other baselines.

Reviewer comment (Evaluation):
Macro-F1 is also not particularly clear. It summarizes precision and recall across the 4 different bases and averages them. Since the problem of inverse folding is effectively a classification problem, other more understandable metrics can be used, such as auROC and auPRC (or visualizing the ROC curve and PR curve across different methods). Again, these can be stratified by secondary structure or tertiary attributes.

Author response:
We are deeply appreciative of the prompt response from the reviewers. 

Our work is not focused on achieving the most advanced secondary structure design, but rather on tertiary structure design. Including additional secondary structure design is beneficial. Given the time constraints, we will endeavor to incorporate additional methods for secondary structure design in our final manuscript. 

In the main text and the appendix, we present comparative analyses across three datasets on five key metrics: recovery, Macro F1, perplexity, AUROC, and AUPRC, thereby offering comprehensive experimental insights. We believe the stratification by structural attribute presents an excellent opportunity for future research.

Considering the efforts invested during the rebuttal stage, we kindly inquire whether you would consider increasing the score. We would be very grateful for your positive feedback.

Reviewer comment (Experiments):
The process of allocating the training/validation/test sets certainly took into account structural similarity by ensuring that different clusters of structures were allocated to entirely different sets. However, the clusters were generated pretty conservatively (i.e. TM score < 0.45, which is already a very low cut-off). This definition of clusters might be useful for contrastive learning, but it might still let a lot of similar structures exist in both the training and test set. Instead of reusing the same clusters, it would be good to ensure that the TM scores between training/test sets are much more significantly different (compared to connected components of TM < 0.45), or splitting the training/test sets based on Rfam families to ensure distinct folds. An analysis that shows minimal cross-contamination of folds/structures/scaffolds between the training and test sets would help put this concern to rest.

Author response:
**A3** Thank you for pointing out this issue. Data leakage is a longstanding problem in bioinformatics. This is why we strictly split the datasets and provide a detailed dataset analysis in **Appendix H** of the manuscript. We believe our dataset is of high quality and free from any data leakage.

The threshold for TM-Score (0.45) is a common choice in RNA structure prediction research, as proposed by Yang Zhang[5]. In their paper published in Nature Methods, the 0.45 cutoff has been demonstrated to effectively distinguish RNA structures.

Reviewer comment (Theory):
Equation 5 is not clear, and does not seem to match typical definitions of MPNNs; for example, it is unclear what the brackets mean and what $h_{E_{ij}}$ is doing on the left; also, it should be $\mathcal{N}(i, K)$ (capital $K$).

Author response:
**A4** Thank you for your careful review. We have refined the manuscript to make it more clear. In Equation 5, the square brackets denote the concatenation operation. The order of the three types of embeddings is not important; here, the edge features between i and j, the node features of i, and the node features of j are concatenated together.

Reviewer comment (Experiments):
Would sharpening confidence be helpful in non-paired residues, as well? It seems like that regularization was added to the paired residues, and the ablation study showed that it was helpful. Would it also be helpful if applied to all residues (not just the paired ones)?

Author response:
**A6** Thank you for your comment. For these non-paired residues, there's no prior knowledge available that could be leveraged to utilize confidence sharpening as a decision-making tool effectively. This distinction in representation between paired and non-paired residues is a key factor in our decision to apply confidence sharpening selectively.

Reviewer comment (Novelty):
The authors should demonstrate a tangible application for their proposed method. Is it intended for designing RNA aptamers? If so, how and why would it be superior to SELEX methods?

Author response:
**Q1** *Tangible application of RNA design*

**A1** Although for many RNAs, the function is not determined by their tertiary structure, there are some applications for tertiary structure-based RNA design. RNA aptamers are good examples that can bind to specific targets with high affinity and specificity. The SELEX process is a traditional method for identifying aptamers. Our approach may provide several advantages as outlined below:

(i) Data efficiency. Our method proposes a data-efficient way to model RNA structures, particularly beneficial for aptamer design due to the complexity of RNA structures and the vast potential sequence space.

(ii) Structural Precision: By focusing on tertiary structures, our approach can provide a more precise way to design aptamers that specifically fit their targets, potentially outperforming SELEX in precision and fit.

(iii) Speed and Scalability: Computational methods can significantly reduce the time and resources needed for designing effective aptamers compared to iterative lab-based methods such as SELEX.

We believe our approach could be a good alternative to SELEX in scenarios where the traditional method is time-consuming and labor-intensive. Thank you for your valuable and professional suggestions.

Reviewer comment (Presentation):
Figure 4 could be made more clear; the green arrows appear too small. Additionally, clusters and samples within clusters should be better aligned.

Author response:
**Q2** *Figure and sentence issues*

**A2** Thanks for you kind suggestion. We have updated the figure in the revised manuscript. The sentence you pointed out should be connected with the first sentence of the following paragraph. We have corrected the paragraphing in the revised manuscript.

Reviewer comment (Experiments):
The base atoms are masked; however, purines (R or A/G) and pyrimidines (Y or C/T) differ significantly in size, and the model could have simply learned to predict the base using the spacing between the backbone atoms of neighboring nucleotides. This is not as significant of an issue in proteins, as there are twenty different side chains that do not canonically pair with each other. One way to test this is to evaluate performance by converting all A/G nucleotides to R and all C/T nucleotides to Y, then report recovery scores based on R and Y only. If this score is substantially higher than the original one, then the model may simply be learning whether bases are purines or pyrimidines.

Author response:
**Q3** *Atom Spacing Significance*

**A3** In our method, we also consider the information of pairing. In the ablation study, it is observed that the removal of pairing information from the secondary structure results in a slight decrease in performance, but the extent of this decline is relatively limited. Furthermore, StructGNN and GraphTrans, two similar structure-based methods, exhibit poorer performance. Even though such issues might exist, we believe their impact is not significant.

Reviewer comment (Experiments):
When constructing the neighborhood graph, wouldn't it be more intuitive to use a ball around each nucleotide? Could enforcing a fixed K result in some edges that are too distant in space?

Author response:
**Q4** *Wouldn't it be more intuitive to use a ball around each nucleotide?*

**A4** In our opinion, using r-ball is similar to k-nearest graph. However, k-nearest is a more convinent way because it always return fix-sized neighbors.

Reviewer comment (Experiments):
Can't this method be iterative? That is, determine the primary sequence from the structure, then compute secondary and tertiary structures from the inferred primary sequence, identify discrepancies, and predict a new primary sequence, repeating this process until a certain condition is met.

Author response:
**Q5** *Can't this method be iterative?*

**A5** This method can be easily adopted into an iterative version. In this paper, we identify the key issue for RNA tertiary structure design problem is the sacarce data and we aim to claim that our data-efficient learning manner is beneficial to this problem. The iterative version is supposed to be useful, and we will try to include it in the future work. Thank you for your valuable suggestion!

Reviewer comment (Evaluation):
Related to the previous question, is the "recovery score" the best metric? Is the goal to predict THE primary sequence or A primary sequence that gives rise to a very similar tertiary structure? Recovery scores, in general, tend to be low, and reporting predicted RMSD (e.g., using RhoFold) might be helpful in this regard.

Author response:
**Q6** *Reporting predicted RMSD (e.g., using RhoFold)?*

**A6** Thank you for highlighting this aspect. Computing tertiary RNA structures from given sequences is indeed computationally intensive. For instance, assessing over 350 RNA structures across three test sets could potentially require upwards of 1050 hours for completion. Additionally, the current scarcity of available RNA tertiary structures, which serve as labels, introduces an element of bias in RNA structure prediction methods, potentially impacting the accuracy of metrics like RMSD.

Given these constraints and considering our primary objective of accurately predicting RNA sequences, employing the recovery score and macro-f1 score as our main performance metrics are both practical and relevant. These scores directly measures how well our model can regenerate the target RNA sequences, making it a suitable and efficient method to evaluate our model's performance under the current computational and data limitations.

Reviewer comment (Experiments):
Approximately 90% of RNA sequences are short (under 100 nt), and a Bayesian Optimization method might work well for this problem. It involves obtaining the tertiary structure, computing the secondary structure, starting with a random sequence, computing its secondary structure, and using the model to propose a new sequence until a close enough solution is found.

Author response:
**Q8** *Bayesian Optimization for RNA Sequences*

**A8** We always enjoy discussing with professional peers like this reviewer. The idea you presented is impressive. It leverages the method's ability to efficiently search complex spaces to find RNA sequences that correspond to a desired tertiary structure. However, its effectiveness would depend on various factors, including the accuracy of the computational models and the availability of sufficient computational resources. Our proposed approach is likely to be more effective than Bayesian Optimization for capturing the complex geometries and dynamic folding patterns of RNA.

Reviewer comment (Evaluation):
Can Cohen's Kappa be used instead of or in addition to Macro F1?

Author response:
**Q9** *Can Cohen's Kappa be used instead of or in addition to Macro F1?*

**A9** Thanks for your valuable comment. Cohen's Kappa is commonly used to assess the degree to which two raters or measurement methods agree beyond chance. In the context of RNA design, where the primary goal is to assess the accuracy of nucleotide-level predictions, the recovery score and Macro F1 are more direct measures of performance as they are designed to evaluate the accuracy of multi-class predictions, which is essential in the context of RNA sequence prediction.

Indeed, using Cohen's Kappa instead of Macro F1 could potentially offer insights into the consistency of predictions in a different way, focusing on agreement or consistency of the categorization process. However, it might not be as directly relevant to the specific task of RNA design. Therefore, while Cohen's Kappa could provide additional insights, especially in terms of consistency or reliability of predictions, the recovery score and Macro F1 seem more directly aligned with the specific goals and nature of RNA sequence prediction tasks.