N orm AL L o RA : What is the perfect size? (2025)

Abstract: Large Language Models (LLMs) are pivotal in enabling intelligent experiences across various applications, from summarization to advanced content organization and retrieval functionalities. However, deploying LLMs for diverse tasks is fundamentally constrained by memory and compute limitations, making it impractical to fine-tune separate models for each task. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) offer a scalable solution for multi-task LLM deployment. Despite its potential, LoRA faces challenges in selecting optimal ranks and layers for each task-model pair, often resulting in inefficiencies and unnecessary parameters. We introduce Norm Adaptive Localized (NormAL) LoRA, a novel variant that employs rank-norm regularization to dynamically determine the optimal rank for each weight matrix, ensuring adaptation is concentrated where it is most impactful. Our approach reduces adapter parameters by 37% while preserving full fine-tuning performance, making NormAL LoRA a transformative tool for enabling efficient, scalable, and space-constrained AI deployments across diverse industries and applications.

## introduction the rapid ascent of transformer @xcite based large language models (llms) has reshaped what we thought was possible in natural language processing: from generating coherent paragraphs to answering complex questions, these billion-parameter models continue to push state-of-the-art performance @xcite @xcite . they are increasingly powering intelligent features in various device-based ecosystems, underscoring the need for scalable, on-device llm deployment (rajbhandari et al., 2022; xu et al., 2023b; sahu et al., 1 https://github.com/jingax/normal 2024). yet, their sheer size brings a practical challenge-fine-tuning for a downstream task is both compute and memory intensive, making it difficult for most users to customize these models without massive resources. parameter-efficient fine-tuning (peft) methods offer an elegant solution by keeping the pretrained weights frozen and learning only a small number of additional parameters @xcite . among these, low-rank adaptation @xcite stands out: by factorizing weight updates into two low-rank matrices, it reduces the trainable footprint dramatically while preserving accuracy. but a new question arises: what rank should each adapter use? too low, and the model underfits; too high, and we waste precious budget. worse, different layers often need very different capacities, turning rank selection into an expensive grid search over dozens-or even hundreds of combinations. identifying the optimal rank configuration for different layers manually is computationally expensive @xcite . to address this problem, we propose norm adpative localized (normal) lora to find the optimal rank for each weight matrix. central to our approach is a new rank-norm regularization mechanism, which promotes knowledge concentration within the adapter weights, enabling further compression through effective layer clipping. ## related work we provide a brief summary of categories of peft and specifically low rank adaptation (lora), and then highlight the gap our work addresses. the peft methods are primarily categorized into: additive modules: prompt-tuning prepends a small set of trainable "soft prompts" to the model input while keeping all pretrained weights frozen @xcite . p-tuning v2 @xcite improves scalability to larger models. adapters insert lightweight bottleneck layers between transformer sub-layers, learning task-specific transformations with only a few million extra parameters @xcite . selective updates: selective fine-tuning restricts updates to particular subsets of model parameters. bitfit updates only the bias terms of each layer, reducing trainable parameters to under 0.1% of the original model @xcite . other variants such as layernorm tuning @xcite or attention-only adaptation @xcite achieve high efficiency with lower representational flexibility. low-rank reparameterizations: low rank adaptation @xcite factorizes each weight update into two low-rank matrices, cutting the number of trainable parameters by orders of magnitude with minimal performance degradation. variants such as llama-adapter @xcite and lora++ @xcite have extended lora to multi-modal and multilingual domains. hybrid & dynamic schemes: hybrid methods combine elements of additive and low-rank approaches. prefix-lora jointly learns soft prompts and lora adapters @xcite , while unipelt unifies prompt, adapter, and sparseupdate strategies in a single framework @xcite . more recently, adalora dynamically allocates per-layer adapter rank based on gradientnorm statistics, optimizing parameter utility @xcite . @xcite further proposes fine-tuning by only training residuals of lora weights for improved generalization. among all the techniques listed, lora has emerged as a particularly powerful technique for efficient fine-tuning of pre-trained models, particularly in scenarios where computational resources are limited. it achieves task-specific adaptation by injecting low-rank trainable matrices into the frozen weight matrices of a pre-trained model. this approach allows the model to adapt to new tasks while preserving the original parameters of the base model. mathematically, lora is expressed as: here, w represents the frozen pre-trained weight matrix, while a and b are low-rank matrices of rank r. this formulation effectively introduces a low-rank adaptor of size r (fig. 1 ), enabling fine-tuning by updating only the adapter weights. however, this traditional perspective treats lora as a unified adaptor, which can obscure the distinct contributions of its individual rank vectors towards specific tasks across various layers @xcite . prior studies have applied heuristic pruning to lora adapters-such as those of @xcite @xcite @xcite @xcite -showing that one can significantly reduce trainable parameters with negligible loss in performance; these methods typically rely on either singular-value decomposition to rank adapter subspaces or on gradient-norm metrics to select adapters for removal. ## proposed methodology we propose a reinterpretation of lora by viewing it not as a single low-rank adaptor but as r individual adaptors of size 1. this perspective allows us to decompose the low-rank matrices a & b into rank vectors v a r & v b r , each of size n × 1 and m × 1, respectively. by expressing the adapted model in terms of these rank vectors, we gain a more granular understanding of lora's adaptation process. equation 1 can be reformulated as: then by application of our novel rank-norm regularization the l2 norms of the lora weights are shifted towards one end (top for lora a and left for lora b). we use norm as a surrogate of layer importance and clip/remove part of lora matrix where l2 norms are lower thus obtaining a new smaller rank lora for that particular layer. by treating lora as a collection of rank-1 adaptors (as above), we can better analyze and optimize its performance, particularly in scenarios where computational efficiency and generalization are critical. this allows for an efficient rank reduction methodology by just determining the most optimal subset of rank vectors for both weight matrices. we propose a 2-stage methodology, described below, to effectively determine the most efficient subset of target vectors for the task. we first propose and demonstrate a novel regularization methodology (knowledge concentration regularization) to effectively concentrate learning. this allows us to effectively remove redundant or low-impact rank-1 adaptors through a clipping strategy. ## conclusion in this work, we introduce normal (norm adaptive localized) lora, which dynamically allocates optical rank to each layer depending on its impact on the downstream task. we introduce a novel knowledge concentration regularization technique with multi-constrained clipping mechanisms. the proposed regularization scheme concentrates most of the model's learned information into the early columns of each adapter. this structure allows the remaining columns to be pruned, yielding a much smaller adapter. to comprehensively assess the versatility and generalizability of normal lora, we conduct a thorough evaluation across a diverse range of natural language processing (nlp) tasks, including both natural language understanding (nlu) and natural language generation (nlg), as well as computer vision tasks such as object detection and image classification. under equal average-rank budgets, normal lora matches or exceeds the performance of existing baselines while also reducing run-to-run variance. specifically, normal lora provides a 37% parameter reduction with no performance drop across nlg baselines. our experiments span models of varying sizes, from 86 million to 7 billion parameters, thereby encompassing a broad spectrum of computational complexities. furthermore, we investigate the applicability of normal lora across different model architectures, including encoder-only, decoder-only, and encoder-decoder configurations. this extensive evaluation aims to provide a holistic understanding of normal lora's efficacy and its potential as a universally applicable parameter-efficient finetuning method. by demonstrating its effectiveness across a wide range of tasks and model architectures, we establish normal lora as a robust and versatile technique for adapting large-scale pre-trained models to diverse downstream applications. as the capabilities of llms increase and more on-device solutions are built on top of base llms, normal lora will be crucial in creating the most optimal and effective adapters in an ondevice, highly resource constrained environment.