Shears: Unstructured Sparsity with Neural Low-rank Adapter Search (2024)

Abstract: Recently, several approaches successfully demonstrated that weight-sharing Neural Architecture Search (NAS) can effectively explore a search space of elastic low-rank adapters (LoRA), allowing the parameter-efficient fine-tuning (PEFT) and compression of large language models. In this paper, we introduce a novel approach called Shears, demonstrating how the integration of cost-effective sparsity and a proposed Neural Low-rank adapter Search (NLS) algorithm can further improve the efficiency of PEFT approaches. Results demonstrate the benefits of Shears compared to other methods, reaching high sparsity levels while improving or with little drop in accuracy, utilizing a single GPU for a pair of hours.

## introduction large language models (llms) exhibit impressive capabilities in comprehensive language understanding, as evidenced by their remarkable zeroshot generation across various tasks. however, supervised fine-tuning is often employed to unlock their true potential in real-world applications. fine-tuning is essential for tailoring performance to domain-specific or proprietary data, bridging the gap between general language understanding and task-specific precision. recently, parameterefficient fine-tuning (peft) @xcite has emerged as a crucial strategy for efficiently boosting the performance of llms in domain-specific tasks. in addition to fine-tuning, increasing model parameters is another critical strategy to improve model performance. llms have produced impressive achievements as they scale to significant sizes, such as palm with 540 billion parameters @xcite . the projection for future models suggests a continuous escalation in parameter * co-first authors. count, anticipating improved performance. however, this trend also underscores the growing demands on computing devices. as model parameters increase, so does the computational complexity, necessitating more powerful hardware and infrastructure. in this context, the importance of model compression becomes particularly evident. model compression is a crucial strategy to mitigate these challenges and make llms more accessible and deployable across a broader spectrum of devices. motivated by the significance of peft and model compression, this paper introduces a novel approach called shears, showing the effective integration of peft and model compression to optimize the llm performance with a high sparsity level. in the proposed methodology, we initiate the process by employing a zeroth-order sparse approach to induce sparsity in the llm. subsequently, we introduce elastic low-rank adapters into the sparsified model and apply neural low-rank adapter search (nls) to train a super-adapter network. finally, a search algorithm is employed to identify the optimal adapter configuration. the contributions of this work can be summarized as follows: the content of this paper uses the following outline: section 2 discusses the algorithms shears uses. section 3 describes the three stages and details of our practical solution. section 4 presents results with several models on a variety of tasks. we offer concluding remarks in section 5, and due to space limitations, we provide more details and a related work section in the appendix. ## methodology in this section, we delve into the proposed approach, shears. figure 1 illustrates the overview of the shears pipeline. as depicted in the figure, the method comprises three key steps: i) unstructured sparsification, ii) super-adapter training, and iii) sub-adapter search. through these steps, the model undergoes sparsification and neural lowrank adapter search while preserving a performance comparable to the fine-tuned model from the original model. next, we discuss the details of each step. ## conclusion this paper presents shears, a practical and novel solution for real-world applications to sparsifying weight-sharing super-networks of elastic adapters (super-adapters). by incorporating elastic lora adapters into the sparsified base model, shears can fine-tune llms without sacrificing the sparsity obtained from the original model weights and produces sparse models with improvements or minor drops in accuracy and a fraction of the cost compared to other approaches. the increase in sparsity can result in significant speedup when using runtimes that take advantage of these patterns. ablation studies show that combining sparsified models with elastic low-rank adapters yields better results than using lora adapters alone. models and code are available at https://github.com/intellabs/hardware-awareautomated-machine-learning .