GRNF ormer: A Biologically-Guided Framework for Integrating Gene Regulatory Networks into RNA Foundation Models (2025)

Abstract: Foundation models for single-cell RNA sequencing (scRNA-seq) have shown promising capabilities in capturing gene expression patterns. However, current approaches face critical limitations: they ignore biological prior knowledge encoded in gene regulatory relationships and fail to leverage multi-omics signals that could provide complementary regulatory insights. In this paper, we propose GRNFormer, a new framework that systematically integrates multi-scale Gene Regulatory Networks (GRNs) inferred from multi-omics data into RNA foundation model training. Our framework introduces two key innovations. First, we introduce a pipeline for constructing hierarchical GRNs that capture regulatory relationships at both cell-type-specific and cell-specific resolutions. Second, we design a structure-aware integration framework that addresses the information asymmetry in GRNs through two technical advances: (1) A graph topological adapter using multi-head cross-attention to weight regulatory relationships dynamically, and (2) a novel edge perturbation strategy that perturb GRNs with biologically-informed co-expression links to augment graph neural network training. Comprehensive experiments have been conducted on three representative downstream tasks across multiple model architectures to demonstrate the effectiveness of GRNFormer. It achieves consistent improvements over state-of-the-art (SoTA) baselines: \mathbf{3.6\\\%} increase in drug response prediction correlation, \mathbf{9.6\\\%} improvement in single-cell drug classification AUC, and \mathbf{1.1\\\%} average gain in gene perturbation prediction accuracy.

## introduction recent advances in foundation models (fms) for single-cell rna sequencing (scrna-seq) analysis has revolutionized our ability to decipher cellular states and gene expression patterns. models like scgpt @xcite , geneformer (theodoris et al., 2023), and scfoundation @xcite demonstrate remarkable capabilities in capturing transcriptomic relationships through large-scale pretraining on millions of cells. these models achieve state-of-the-art performance in critical tasks, including cell type annotation, perturbation prediction, and multi-omic integration. particularly noteworthy is scpalm @xcite , which introduces biological pathway-aware representations to address computational challenges in transformer-based approaches. however, despite their successes, current rna fms face fundamental limitations rooted in their reliance on expression data alone. three key challenges persist in existing approaches. first, as shown in fig. 1 , while current models learn genegene correlations implicitly, they lack explicit integration of regulatory causality derived from chromatin accessibility data -a crucial determinant of cellular identity @xcite . second, existing methods struggle to capture the multi-scale nature of gene regulation, where relationships operate at both cell-type-specific and cell-specific @xcite . third, se-vere information asymmetry plagues regulatory networks: for some cell types, transcription factors (tfs) exhibit dense connectivity while ∼ 40% of genes lack reliable regulatory links @xcite , creating a topological imbalance that standard architectures cannot effectively handle @xcite . to handle these challenges, we present grnformer, a novel architecture that integrates multi-scale gene regulatory networks (grns) into current rna fms through three key innovations. first, we introduce a systematic pipeline for constructing cell-specific and cell-type-specific grns capturing regulatory causality derived from chromatin accessibility data by integrating singlecell atac-seq (scatac-seq) and scrna-seq data utilizing scenic+ @xcite . as shown in fig. 2a and appendix a, our method leverages chromatin accessibility to identify enhancer-driven regulatory units (eregulons) through motif enrichment analysis and multimodal linkage @xcite , enabling discovery of context-specific regulatory relationships across biological scales. then, we introduce a universal structure-aware integration framework that utilizes the multi-scale gene regulation information and addresses grns topological challenges through: i) an adaptive cross-attention layer that dynamically weights regulatory signals based on node centrality and ii) a biologically informed edge perturbation strategy that supplements sparse connections with coexpression relationships as shown in fig. 2c . this design enables effective knowledge transfer from grns while mitigating information asymmetry -a critical advancement over naive fusion approaches such as addition or concatenation. lastly, we establish comprehensive benchmarks across three clinically-relevant tasks: gene perturbation prediction, drug response classification, and single-cell sensitivity analysis. our experiments demonstrate that grnformer achieves consistent improvements over base models (scgpt +3.6% pearson correlation coefficient (pcc) on the drug response prediction task, scfoundation +4.1% area under the roc curve (auc) on the single cell drug response classification task). notably, the model reveals interpretable attention patterns aligning with known biological regulations. our key contributions are three folds: the first systematic framework integrating scatac-seq and scrna-seq data to build celltype-specific and single-cell-resolution regulatory networks through enhancer-driven eregulons analysis pipelines. tf motif enrichment analysis annotated tf-motif collection enhancer-driven grn inference cell-specific grn ...... ...... cell-type-specific grn ...... candidate enhancers scrna-seq scatac-seq motif database curated motif-to-tf annotations tf-cell activity score binarized activity score motif clustering perturbed cell-type-specific grn perturbed cell-specific grn cell-specific grn shared gnn layers with fixed node sampling cross attention fusion layer ...... ...... ...... ## experiments we conducted extensive experiments to evaluate grnformer across three biologically significant tasks: ❶ gene perturbation prediction examines the model's ability to capture regulatory mechanisms by predicting gene expression changes following gene perturbations. this task is particularly relevant for therapeutic development and understanding disease mechanisms. ❷ drug response prediction evaluates the model's clinical utility by predicting cellular responses to therapeutic compounds. the model integrates gene expression profiles with drug structural information to predict ic50 values (half-maximal inhibitory concentrations). ❸ single-cell drug response classification tests the model's ability to transfer knowledge from bulk cell line to single-cell resolution, a critical capability for personalized medicine. the task involves predicting drug sensitivity for individual cells. across all these tasks, we will compare our approach against sota baselines and conduct comprehensive ablation studies to evaluate our grn integration strategy's effectiveness systematically. this multi-faceted evaluation framework ensures a thorough assessment of our approach's biological accuracy and practical utility. ## conclusion in this paper, we propose grnformer, a framework that systematically integrates multi-scale gene regulatory networks into rna foundation models through two key innovations: (1) hierarchical grn construction via multi-omics fusion, and