In Defense of Structural Sparse Adapters for Concurrent LLM Serving (2024)

Abstract: Adapting large language models (LLMs) to specific tasks remains challenging due to the extensive retraining required, prompting the need for efficient adapter techniques. Despite this, the concurrent serving of multiple adapters, each with unique matrix shapes, poses significant system-level challenges. To address these issues, we identify an opportunity in structurally sparse adapters, which, unlike low-rank adapters, maintain consistent matrix shapes while varying in sparsity patterns. Leveraging this characteristic, we introduce SpartanServe, a system designed for efficient concurrent serving of LLMs using multiple structurally sparse adapters. SpartanServe employs a unified matrix multiplication operation and a novel memory management technique to enable effective batching. Furthermore, the incorporation of Triton kernels enhances the acceleration of matrix multiplication in the serving process. Experimental results demonstrate that SpartanServe achieves 2.12× speedup over S-LoRA when serving 96 adapters using a single NVIDIA A100 GPU (40GB), showcasing its efficacy in concurrent LLM serving.

## introduction as the field of natural language processing (nlp) continues to advance, large language models (llms) @xcite have emerged as a cornerstone technology, powering a wide range of applications from automated customer service @xcite to sophisticated content generation @xcite . the need for adapters. large language models (llms) require adapters like low-rank adaptation (lora) @xcite to efficiently finetune their performance on specific tasks while managing computational and resource constraints. llms, with their extensive parameters and capabilities, still face challenges in adapting to new or niche applications without exhaustive retraining. adapters offer a solution by introducing lightweight, task-specific adaptations that modify only a subset of the model's parameters @xcite @xcite . this approach significantly reduces the memory and computational power needed for fine-tuning, making it feasible to deploy llms in varied and resource-limited environments. expensiveness of multi-adapter concurrent llm serving. with the rise of llm adaptation techniques, serving multiple adapters on the same base llm has become standard practice. as a consequence, finding efficient methods to serve several adapters concurrently has become an interesting research area. one intuitive approach to efficient adapter serving is batching the input requests and their corresponding adapters for faster inference. however, this concurrent serving of multiple adapters is costly due to the irregularity of the adapters. specifically, adapters like lora require specifying the hidden low-rank dimension as a hyper-parameter, resulting in different adapters having different low-rank matrix shapes. to address this issue, @xcite proposed s-lora, which tackles the heterogeneous shapes of low-rank matrices. despite these significant system-level improvements, challenges persist in concurrently serving llms with numerous, heterogeneous lora adapters. an opportunity from structural sparse adapters. in this work, we identify an opportunity for efficient adapter batching presented by structurally sparse adapters @xcite . we argue that there is a fundamental difference between structurally sparse adapters and low-rank adapters: for the same large language model (llm), structurally sparse adapters maintain the same matrix shape but exhibit different sparsity patterns. leveraging this advantage, we propose a unified matrix multiplication approach for concurrent llm serving. ## conclusion in this work, we study the concurrent llm serving with multiple adapters. we have explored the potential of structurally sparse adapters, which maintain consistent matrix shapes while varying in sparsity patterns, unlike low-rank adapters. this insight led to the development of spartanserve, a system designed for the efficient concurrent serving of llms using multiple structurally sparse adapters. spar-tanserve leverages a unified matrix multiplication operation and an innovative memory management technique to enable effective batching. additionally, the use of triton kernels enhances the acceleration of matrix multiplication during the serving process. experimental results demonstrate that spar-tanserve achieves 2.12× speedup over s-lora when serving 96 adapters using a single nvidia a100 gpu (40gb).