MP - RNA : Unleashing Multi-species RNA Foundation Model via Calibrated Secondary Structure Prediction (2024)

Abstract: RNA foundation models (FMs) have been extensively used to interpret genomic sequences and address a wide range of in-silico genomic tasks. However, current RNA FMs often overlook the incorporation of secondary structures in the pretraining of FMs, which impedes the effectiveness in various genomic tasks. To address this problem, we leverage filtered high-fidelity structure annotations for structure pretraining to enhance the modeling ability of FMs in single nucleotide resolution tasks. Experimental evaluations across four comprehensive genomic benchmarks demonstrate that our RNA FM consistently outperforms existing RNA FMs, achieving a 40% improvement in RNA secondary structure prediction and obtaining top-tier results on DNA genomic benchmarks even though it has not been pretrained on any DNA genome. We release the code and models to encourage further research to bridge the gap between in-silico predictions and biological reality.

## introduction rna serves as a critical molecule in a variety of important cellular processes and controls the flow of genetic information from dna to protein @xcite . with the development of highthroughput rna sequencing @xcite , understanding the vast of rna composed of nucleotide sequences reaches the efficiency and performance bottleneck of bioinformatics techniques. recent studies leveraged genomic foundation models (fms) to understand dna sequences @xcite @xcite @xcite and rna @xcite @xcite @xcite sequences and address a broad spectrum of in-silico genomic tasks such as mrna vaccine design @xcite , translation efficiency prediction @xcite , and gene expression prediction @xcite . this is because the generalization of language modeling from natural sentences to rna sequences in the rna 'language' @xcite is intuitive and the fms can efficiently model genomic sequences. therefore, in the era of high-throughput genome sequencing, fms are the vital bridge to learning intriguing genomic information from the tremendous rna and dna sequence databases. despite preliminary results of the previous dna and rna fms to handle diverse genomic tasks @xcite @xcite , existing fms usually struggle in challenging genomic tasks, e.g., single nucleotide resolution tasks, hindering wide acknowledgment from bio-scientists and wide applications of fms. since it has been acknowledged that "the functionality and stability of rna significantly depend on complex structures, e.g., secondary structures, in molecular biology" @xcite , this difficulty is probably because current fms overlooked rna secondary structures and straightforwardly took the techniques from language modeling to rna language learning, leading to unimpressive performance on various genomes types (e.g., rna and dna, etc.). moreover, this incapability in existing fms limits the transferability to diversified unknown species because the genomes of millions of species are technically represented in numerous genomic 'language' variants, which is very different from the natural language modeling area. as the existing fms struggle with challenging genomic tasks, we aim to propose a new multi-species plant rna (mp-rna) foundational model and prove it is robust and effective in addressing universal rna genomic tasks. we elaborate on the main challenges that we met and our solutions one by one. the first challenge is to leverage secondary structures in rna modeling. learning rna secondary structures is extremely challenging @xcite because the sequence information is generally sparse @xcite and structure candidates of specific sequences grow exponentially with increasing sequence lengths. besides, an identical sequence may fold into different secondary structures because the folding principles of rna sequences depend on a variety of biological factors @xcite , such as dynamics mechanism @xcite ). we show an example in fig. 1 that two secondary structure variants belong to an identical rna sequence. one simple but effective method to mitigate this problem is to exploit large-scale structure annotations @xcite @xcite to supervise the fms in structure prediction. unfortunately, we cannot obtain enough rna secondary structure annotations for pretraining because the structure annotations are complicated and expensive @xcite . to compile a largescale database containing both sequence-structure pairs for fm pretraining, we utilize a well-known secondary structure prediction (ssp) tool, vien-narna @xcite , to obtain millions of plausible secondary structures with corresponding sequences. we show that pretraining on millions of secondary structures yields an effective and robust ssp, and it can improve the performance on various rna modeling tasks even without structures involved as demonstrated in our experiments. the second challenge of structure pretraining comes from the low fidelity of the secondary structure annotations. although the secondary structure annotations calculated by viennarna improve our fm's performance on extensive in-silico experiments, our study reveals a substantial problem, i.e., the secondary structures based on calculation, such as minimum free energy @xcite , fail to reflect the genuine in vivo structures. for example, the structure prediction f1 score of vien-narna on the bprna @xcite dataset is only ≈ 28%. the misalignment between vast sequences and plausible secondary structures casts a shadow on the structure pretraining, as well as various downstream genomic tasks. although it is impossible to collect large-scale biologically verified structures at this stage, we realize that the fm, fine-tuned on the formal secondary structure datasets @xcite , yields the structures almost there, i.e., close to the genuine structures. additionally, we have to make sure the structure predictions can be trusted before we employ them, so we calibrate the predictions from the perspective of uncertainty and perform incremental pretraining to enhance the fm. we adopt the popular temperature scaling @xcite , as it does not influence the model's performance, to calibrate the predicted structures as high-fidelity structure annotations to incrementally pretrain the fm and refine structure modeling ability. the incremental pretraining uses the top 10% of the rna samples compared to the first pretraining to enhance the fm. the experimental results show that incremen-tal pretraining comprehensively improves structure prediction tasks as well as four comprehensive genomic benchmarks. the third challenge is to refine the rna sequence modeling granularity, as our study indicates that the existing fms are usually designed for sequence-level tasks, e.g. sequence classifications. however, most of the in vivo genomic analysis focus on single nucleotide resolution, such as single nucleotide variant @xcite ) (snv) and single nucleotide polymorphism @xcite (snp), mutation detection and repair @xcite . besides, single nucleotide (sn) base changes can influence rna folding results @xcite , structure stability and molecular functionality, etc. to address this obstacle, we include single nucleotide mutation repair in the pretraining objectives, i.e., we pretrain the fm to find the mutation sites and predict the original bases in the rna sequences. this sn mutation modeling is a pioneering effort to understand the dynamic connections between rna sequences, structures, and sn mutations @xcite . to benefit future research of rna fms, we will release the pretrained models, benchmarks, and in-silico rna modeling cases in this work. ## evaluations and findings research on rna fms remains in a state of beginning, as the related theoretical and empirical studies are far from comprehensively completing the jigsaw of genomic sequence modeling. we have implemented comprehensive benchmarks to evaluate mp-rna. as insights for future works, we summarize our findings as follows: our empirical experience indicates that adopting mixed tokenizations based on the existing tokenizations may balance the performance and efficiency of multi-granular tasks. ## methodology in this section, we delve into the methodology of our pretraining paradigm step by step. ## related works the field of biological sequence modeling, encompassing dna, rna, and proteins, has garnered increasing interest over recent years. while protein modeling has been extensively researched, as evidenced by projects such as alphafold @xcite @xcite and esm @xcite , dna and rna modeling have seen relatively less exploration. among the earlier efforts in genomic sequence modeling, dnabert @xcite ) utilized bert's architecture @xcite to tackle genomic sequence analysis, demonstrating initial success for in-silico genomic tasks. its successor, dnabert2 @xcite , enhanced this approach by switching from k-mers to bpe tokenization, aiming to boost model performance across multi-species genomic data. exploring large-scale foundation models (fms), such as nucleotide transformers (v1 and v2)(dalla-torre et al., 2023), agront (mendoza-revilla et al., 2023), and segmentnt (de almeida et al., 2024), has proven fruitful. these models, with parameters in the billions, have shown considerable promise in dna genomic modeling, handling model scales up to 2.5 billion and 1 billion parameters, respectively. although agro-nt @xcite was initially pre-trained on multi-species edible plant dna sequences, it did not effectively adapt to rna sequence modeling in subsequent tests. the challenge posed by the extensive lengths of genomes has increasingly shifted focus towards long-range sequence modeling and the deployment of autoregressive fms like hye-nadna @xcite and evo @xcite . in rna genomic modeling, early endeavors such as scbert @xcite , rn-abert @xcite , rna-fm @xcite , rna-msm @xcite , and rnaernie @xcite have emerged. these models, however, have only been trained on limited-scale databases due to the high cost of acquiring rna sequences. some focus narrowly on specific rna sequence types like coding sequences (cds) @xcite , 5' untranslated regions (5'utr) @xcite , 3' untranslated regions (3'utr) @xcite , or precursor mrna sequences @xcite , restricting their ability to capture the full diversity of rna sequences. despite reports of effective performance, uni-rna @xcite remains closed-source and thus unverifiable in comparative experiments. overall, existing fms often overlook the critical issue of calibrated structure pretraining in rna genomic modeling. while 5utr-lm @xcite and ernie-rna (yin et al., 2024) integrate secondary structures into genomic modeling, they do not investigate the effects of the structure annotation quality. ## open-source package genomic modeling is at a nascent stage. there is a domain knowledge gap between artificial intelligence and bio-science which results in a significant scarcity of research resources. although some works for genomic sequence modeling have been proposed in recent years. however, most of these studies only release the model weights without accompanying training, fine-tuning, and benchmark evaluation scripts. to solve this problem, we have created a comprehensive open-source genomic modeling toolkit based on mp-rna. this toolkit is designed to offer thorough fine-tuning tutorials and a standardized automated benchmark evaluation system. please find the brief introduction of the package in appendix a. ## conclusion our work focused on the refinement of rna fms through the rigorous integration of domain-specific knowledge, particularly secondary structures and single nucleotide mutations, into the pretraining phase. the introduction of a calibration mechanism using temperature scaling further enhanced the accuracy and reliability of our model predictions, aligning them more closely with biological realities.