ORAL : Prompting Your Large-Scale L o RA s via Conditional Recurrent Diffusion (2025)

Abstract: Parameter generation has emerged as a novel paradigm for neural network development, offering an alternative to traditional neural network training by synthesizing high-quality model weights directly. In the context of Low-Rank Adaptation (LoRA) for evolving ( i.e, constantly updated) large language models (LLMs), this approach promises efficient adaptation without costly retraining. However, existing methods face critical limitations in simultaneously achieving scalability and controllability. In this paper, we introduce \texttt{ORAL} , a novel \textit{conditional recurrent diffusion} framework that addresses these challenges. \texttt{ORAL} incorporates a novel conditioning mechanism that integrates model architecture and textual task specifications, enabling the generation of task-specific LoRA parameters that can seamlessly transfer across evolving foundation models. Our approach successfully scales to billions-of-parameter LLMs and maintains controllability. Through extensive experiments across seven language tasks, four vision tasks, and three multimodal tasks using five pre-trained LLMs, we demonstrate that \texttt{ORAL} generates high-quality LoRA parameters that achieve comparable or superior performance to vanilla trained counterparts.

## introduction recent advancements in generative ai have been fueled by the vast amount of valuable data throughout the internet, demonstrating profound transformations in the creation of texts, images, and videos @xcite @xcite @xcite @xcite . inspired by these breakthroughs, researchers have begun to investigate whether the vast number of online pretrained model checkpoints can serve as an untapped resource for ai development. building on weightspace learning @xcite ## experiments in this section, we will evaluate our conditional large-scale lora generation method to generate parameters across multiple tasks, models, and modalities. we first report our experimental setup that is constant across all experiments, like training details, and then experiment-dependent settings and results are contained within their own sections. ## conclusion we introduced oral, a novel framework for conditional recurrent diffusion that enables scalable generation of low-rank adaptation (lora) parameters at large scale. our approach is designed to address the critical challenge of adapting to constantly evolving large language models without the computational burden of retraining. by incorporating both model architecture and textual task specifications as conditional inputs to the diffusion process, oral achieves remarkable flexibility and portability across model architectures while maintaining performance comparable or superior to traditionally trained lora parameters. our comprehensive experiments across various diverse tasks, three state-of-the-art pre-trained models (stable-diffusion2.1, mistral, qwen-7b-vl), and two modalities demonstrate the practical viability of our approach for real-world deployment. oral successfully scales to generate hundreds of millions of parameters while preserving adaptation quality. in the future, exploring the interpretability of generated parameters could provide insights into how different tasks influence parameter properties.