Low-Rank Interconnected Adaptation across Layers (2025)

Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates \Delta W = AB for pretrained weights W through low-rank adapters A and B . While LoRA ensures hardware efficiency, its low-rank weight updates limit adaptation performance. In this paper, we propose low-rank interconnected adaptation across layers (Lily), a novel PEFT method that introduces an interconnected framework with locally shared A and globally shared B experts. This structure eliminates redundant per-layer AB pairs, enabling higher-rank \Delta W with equal or fewer parameters. To enhance expressiveness, we use data-dependent routers to determine A - B interconnections, preventing B experts from converging to the same behavior and improving representational power across domains. Experiments across modalities, architectures, and model sizes demonstrate Lily’s superior performance and efficiency.

## introduction fine-tuning foundation models like transformers @xcite on downstream tasks is common but costly, especially for large models like llms, which incur high computational and storage demands and risk catastrophic forgetting @xcite . linear probing alleviates these issues by fine-tuning only the final modules, but suffers from performance loss due to frozen backbone weights. to address this, parameter-efficient fine-tuning (peft) freezes the backbone and introduces lightweight modules for task-specific learning. among peft methods, low-rank adaptation (lora @xcite ) is widely used, particularly for llms. lora introduces low-rank projection matrices, a and b, to approximate weight figure 1 : dynamics of lora and lily. in this 6-layer example with a fixed overall parameter budget, lora allocates the same parameter budget to each layer, resulting in small rank updates for the weights. lily overcomes this by employing a small number of shared adapters with a much larger rank, achieving higher-rank updates while using the same or even a smaller parameter budget. considering the different characteristics, and to make the adaptation more dynamic, the adapters are mixed according to a data-dependent router, represented by r. however, lora and its subsequent improvements @xcite @xcite face a limitation: the learned weight updates ∆w are constrained to be low-rank, limiting model performance. a key issue is that lora allocates the same parameter budget to each layer, regardless of their importance (fig. 1 ). as a result, the rank of each adapter is constrained by the fixed budget, raising the critical question: can we enable more dynamic, expressive adaptation with high-rank weight updates under the same parameter budget? in this paper, we propose low-rank interconnected adaptation across layers (lily), a novel framework for more expressive and efficient peft. specifically, we decouple a and its corresponding upward b, eliminating their tight coupling. each a is connected to all bs, and vice versa, as illustrated in fig. 1 . this creates a hierarchical structure where locally-shared as perform downward projections at specific layers, while globally-shared bs perform upward projections across all layers. to enhance dynamism, we selectively connect each a with bs based on layer features. a extracts features from the current layer, and a selective mixture of bs is performed based on these features, enabled by routers @xcite that generate data-dependent weight distributions for b experts. the interconnected structure makes the adaptation process more dynamic and flexible, with rich interactions between adapters. by reducing the number of adapters and increasing their rank, lily achieves higher-rank weight updates than lora while using the same or fewer parameters. additionally, lily enables comprehensive information access and learning by allowing adapters at each layer to collaborate, share knowledge, and model dependencies across layers. our key contributions include: parameter efficient fine-tuning foundation models are typically pre-trained on large datasets and fine-tuned on downstream tasks. parameterefficient fine-tuning (peft) seeks to fine-tune models efficiently with minimal parameters while maintaining performance and preserving learned knowledge. it effectively addresses limitations of conventional fine-tuning techniques, like full fine-tuning or linear probing. current peft approaches can be divided into two categories: 1) adapter-based methods @xcite @xcite @xcite and 2) promptbased methods @xcite . adapter-based methods insert lightweight adapters into the multi-head self-attention (mhsa) or feed-forward network (ffn) blocks of the transformer architecture, while prompt-based methods add trainable tokens to the input sequence. among these, low-rank adaptation (lora @xcite ) is a well-known technique. it introduces projection matrices a and b for each adaptation target w , where a projects input x to a low-dimensional space and b restores it to the original dimension. the product of these matrices approximates the weight update ∆w in full finetuning (fft). however, this limits the update to a low-rank subspace, which may affect performance. additionally, a and b are tightly coupled, restricting the adaptation process to information from the current layer, which may hinder the modeling of dependencies across layers. mixture of experts mixture of experts (moe) is an active research area that has received significant attention, especially in the field of large language models (llms). conditional computation, which activates different parts of the network on a per-example basis, has been proposed to enhance model capability without increasing computation @xcite @xcite . the sparsely-gated moe layer is introduced to implement this idea, consisting of numerous subnetworks @xcite . a trainable gating network (router) determines the combination of experts for each example. there are already peft methods like molora @xcite and mola @xcite ) that apply the moe design concept to peft. however, these methods simply treat the adapters a and b combined in lora as a single expert. concurrent research @xcite utilizes a and b sub-spaces as the experts but fails to overcome the limitation discussed in the previous section. another concurrent work, hydralora @xcite , explores an asymmetric design for lora. unlike our work, we consider the interconnection across layers and deploy a model-wide asymmetric design to enable cross-layer connections. this enables the use of adapters of higher rank than a typical lora setup while using the same or fewer overall parameters. ## experiments we validate the effectiveness of lily across different domains, model sizes (from vit to llm), and architectures (transformers, mamba), demonstrating its generally strong adaptation capability. concurrently, we conduct a comprehensive analysis of lily's intrinsic mechanisms, providing a thorough understanding of how it works. all ranks for lily are selected from 8, 16, 32, ensuring that the total parameter count does not exceed that of the baselines. all experiments are conducted on a single rtx 4090 gpu. additionally, multiple analyses are provided in appendix c, d, e, f, g, h, i, and j. ## conclusion in this paper, we propose low-rank interconnected adaptation (lily), a novel framework for efficient fine-tuning via the interconnectivity of adapters. lily enables each layer to access information from others during adaptation through a hierarchical structure. additionally, it successfully overcomes the low-rank update limitation of lora, enabling high-rank updates and, therefore, better adaptation capability under the same parameter budget. our approach consistently improves performance across various modalities, model sizes, and architectures, surpassing existing methods while maintaining enhanced efficiency. in summary, lily's versatility and efficiency make it a promising approach for a wide range of applications.