Paper: Splicing Up Your Predictions with RNA Contrastive Learning (ICLR-2024)

Reviewer comment (Experiments):
The number of datasets is limited. Only three datasets are used, among which the performance on Gene ontology is only partially reported in the Appendix.

Author response:
We aim to improve on this in a further iteration, however, we believe the substantial improvement observed in linear probing results over previous methods has merit.

Reviewer comment (Experiments):
What's the point of learning good RNA representations if all your downstream tasks can be solved with standard fine-tuning? In NLP, better sentence embedding can directly be used to amplify retrieval-based QA systems or retrieval-based text generation. But in your application, I do not see the better embeddings helping your downstream applications.

Author response:
First, we’d like to point the reviewer to Figure 3b in which we demonstrate IsoCLR’s significant improvements in low data availability prediction setting over supervised methods. In biological sciences lower throughput experiments are more common compared to other domains such as language and vision underscoring the utility of this result.

In addition, we demonstrate IsoCLR’s improvement over supervised models with matched architectures in Table 3. These results demonstrate the general utility of our methods highlighting the potential for improvement across a diversity of architectures.

Reviewer comment (Novelty):
If I understand correctly, the proposed method works comparably with `Saluki` in full fine-tuning while performing worse in linear probing than it. It also has more parameters than it. What is the benefit of your model over `Saluki`?

Author response:
Our linear probing results in Table 2 / Figure 1 compare IsoCLR’s linear probing results to full model fine-tuning. For linear probing we do not have to train the full model, optimizing only the last layer to achieve 85% of the Saluki full fine-tuning performance.

This is interpreted as a successful finding in transfer-learning as the original SimCLR paper also does not outperform the fully supervised ResNet in linear probing regime (they match the performance of the 4x smaller model when restricting training to 90 epochs vs 1000 epochs https://arxiv.org/pdf/2002.05709.pdf figure 7). 

In addition, we’d like to point out that we perform significantly better than the existing self-supervised methods.

Reviewer comment (Experiments):
If I understand correctly, the proposed model and `Saluki` utilize more information than the sequence itself. If this is true, then the comparison with the baselines is unfair.

Author response:
Most of the methods we compare against are designed for DNA tasks and so do not possess this RNA-relevant information. We did consider this and wanted to understand whether our performance stems from additional information or the pre-training procedure so we conducted an ablation study. We confirm that our smallest model outperforms all the other self-supervised learning methods on both RNA half-life tasks and Gene ontology and can be found under Splice + Homology + Mask row in Table 4. We outperform the other models on the mean ribosome load task as well which we can add to the appendix.

Reviewer comment (Experiments):
Why do you present the results on linear probing with models like RNA-FM, NT, and DNA-BERT2 while skipping them in the model fine-tuning? If they can generate embeddings for linear probing on the datasets, you should be able to fine-tune them on the tasks, too. The models, except for NT-2.5b, are not too large to be fine-tuned on consumer GPUs.

Author response:
The existing models have a truncated context window relative to the length of RNA sequences. Most methods have 512-1024 context window length whereas to capture 95% of RNA sequences our context length is 12,000 nucleotides in length. During the evaluation of linear probing, we perform a concatenation of average pooling and max pooling to address this issue. It is unclear how to perform this in full-model fine-tuning as the existing context window is between 5-10% of the overall sequence length. We did not perform this experiment since we expected poor results given our margin of performance relative to HyenaDNA. We’ve noted the challenges associated with short context length models in section 4.2 but we will aim to further clarify this in text.

Reviewer comment (Experiments):
For example, RNA-FM (which was compared to isoCLR in experiments) demonstrate its effectiveness in structural-related prediction (secondary structure predictions, RNA contact predictions, 3D distances, etc.), which is a more crucial aspect in the related field. The current submission is mostly focused on relatively easier tasks, which is not sufficient to verify the effectiveness of learned representations.

Author response:
Thank you for a detail-oriented review. Given the overall reviewer sentiment, we have opted not to run experiments for RNA structure prediction for this submission. We agree that including structural prediction tasks will make our evaluations more comprehensive.