SSML o RA : Enhancing Low-Rank Adaptation with State Space Model (2025)

Abstract: Fine-tuning is a key approach for adapting language models to specific downstream tasks, but updating all model parameters becomes impractical as model sizes increase.Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address this challenge by introducing additional adaptation parameters into pre-trained weight matrices.However, LoRA’s performance varies across different insertion points within the model, highlighting potential parameter inefficiency due to unnecessary insertions. To this end, we propose SSMLoRA (**S**tate **S**pace **M**odel **L**ow-**R**ank **A**daptation), an extension of LoRA that incorporates a State Space Model (SSM) to interconnect low-rank matrices. SSMLoRA ensures that performance is maintained even with sparser insertions. SSMLoRA allows the model to not only map inputs to a low-rank space for better feature extraction but also leverage the computations from the previous low-rank space. Our method achieves comparable performance to LoRA on the General Language Understanding Evaluation (GLUE) benchmark while using only half the parameters. Additionally, due to its structure, SSMLoRA shows promise in handling tasks with longer input sequences.

## introduction fine-tuning @xcite is a technique aimed at enhancing model performance by adjusting the parameters of a pretrained model on specific task data. this approach effectively leverages the knowledge accumulated through large-scale training, accelerating the model's adaptation to new tasks and improving results. however, fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. these limitations have prompted researchers to ex-* corresponding author plore more efficient parameter adjustment strategies. to address resource consumption and overfitting challenges in fine-tuning, researchers have proposed diverse strategies. @xcite introduced an end-to-end hierarchical fine-tuning approach to mitigate memory constraints associated with full-parameter fine-tuning. concurrently, a significant research trend has emerged focusing on parameter-efficient fine-tuning (peft) @xcite techniques, which are a set of methods designed to optimize parameter updates. peft updates only a small subset of the model's parameters while keeping the majority of pretrained parameters unchanged. lora @xcite , one of the most commonly used peft techniques, introduces low-rank matrices to adjust specific weights in the pretrained model instead of updating all parameters comprehensively. this allows the model to retain most of its structure while adapting to downstream tasks, particularly for transformer-based deep language models. following the successful application of lora, various extensions have emerged to further enhance its adaptability and efficiency. these methods aim to improve the model's capability to perform well in resource-constrained environments. despite its success, lora and its variants still face certain challenges. sora @xcite has indicated that lora may lead to unnecessary parameter overhead. more efficient methods, such as pruning certain low-rank matrices and using smaller scaling factors to reduce parameter usage, were employed. moreover, compared to full finetuning, parameter-efficient methods tend to have limitations in capturing information, exacerbating the inherent difficulty that transformer-based models face when processing long texts. this leads to the observation that, for many tasks, the number of parameters used by lora is excessive, while for tasks involving long-text processing in large language models, it is insufficient. to address this issue, we propose state space model low-rank adaptation (ssmlora), a novel technique that builds on the state-space equation framework to insert sparse low-rank matrices. we introduce interval layers and apply our technique selectively to the query and value matrices within the attention mechanism, as this design significantly reduces parameter overhead. additionally, we incorporate a state space model to connect lowrank matrices inserted into the same type of neural network layers. state space model, which excels at handling long-text tasks, is based on state transitions but differs from traditional rnns by enabling parallel computation through fft-based transformations of the state transitions. this enhancement facilitates parallel training in our approach. experimental results demonstrate that our technique performs well when applied to both linear and convolutional layers. compared to lora, ssmlora achieves superior performance across a wide range of tasks while requiring only half of the parameters. furthermore, it excels in long-text and certain other specific tasks. we believe that this represents a new direction in the exploration of lora-derived techniques, adopting a more holistic and macro-level perspective. ## conclusion in this work, we propose a novel approach for parameter-efficient fine-tuning of large pre-trained language models, termed state space model low-rank adaptation (ssmlora). while the insertion of low-rank matrices has been a common strategy, it often leads to unnecessary parameter overhead, particularly when applied uniformly across all neural network layers. to address this, we introduce an improved state space equation that strategically connects sparsely inserted low-rank matrices, significantly reducing the number of parameters while enhancing parameter efficiency. our method is simple yet effective, and given the demonstrated potential of state space model, it offers a holistic improvement to lora techniques. furthermore, our approach can be seamlessly integrated with existing optimization strategies, further expanding its applicability in large-scale model fine-tuning.