L o RAE xit: Empowering Dynamic Modulation of LLM s in Resource-limited Settings using Low-rank Adapters (2024)

Abstract: Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing tasks. However, deploying LLMs on resource-limited settings remains a challenge. While early-exit techniques offer an effective approach, they often require compromised training methods that result in sub-optimal performance. On the other hand, multi-model methods achieve improved results but suffer from significant inference latency and memory consumption. In this paper, we propose LoRAExit, a novel dynamic inference architecture that leverages low-rank adaptors for efficient deployment of LLMs. LoRAExit decouples the training of multiple exit interfaces, enabling the separate optimization of each exit, thereby fundamentally addressing the performance issues of early-exit networks. Moreover, we introduce a superior-exit guided distillation method that effectively utilizes models of different sizes, thereby further enhancing the performance of early exits. Experimental results demonstrate that LoRAExit significantly improves LLM performance when deployed on resource-limited settings.

## introduction large language models (llms) have achieved remarkable success in various natural language processing tasks, demonstrating their potential for advancing the state-of-the-art in language understanding and generation @xcite . however, the deployment of llms on resource-limited settings, such as edge devices and systems with constrained computational resources, presents significant challenges @xcite . the computational demands of llms, characterized by their large size and high memory requirements, hinder their efficient deployment in such settings. in recent years, there has been significant research exploring various approaches to enhance the efficiency of llms during inference. one prominent approach is adaptive inference, where computations are dynamically allocated based on data difficulty @xcite . among the effective methods, early-exit networks have garnered attention, enabling predictions at different inference stages and allowing for early termination when confident predictions are obtained @xcite . however, these early-exit techniques often rely on compromised training methods, which can lead to sub-optimal performance. figure 1 illustrates three representative training methods. the joint training method optimizes all parameters simultaneously, while the two-stage training method involves training the llm backbone in the first stage and the intermediate classifiers in the second stage. these methods suffer from conflicting gradient issues that can significantly impact model performance. the local training method mitigates this issue by heuristically distributing each loss to specific layers, attempting to alleviate the conflicting gradients. however, it remains challenging to accurately distribute the loss to specific parameters, and simple heuristics may inadvertently harm performance. another notable approach is multi-model methods, which leverage combinations of models to achieve improved results @xcite . nevertheless, these methods suffer from increased inference latency and memory consumption, posing challenges in resource-limited settings. overcoming these limitations and building an efficient adaptive model present two key challenges. firstly, we need to train a model of different sizes without compromising its performance. secondly, we must ensure that the model can run without significantly increasing the resource requirements. these challenges are inherently conflicting because sharing parameters in early exits can degrade performance, while completely separating parameters will inevitably increase the resource requirements. to tackle these challenges, we propose lo-raexit, a novel dynamic inference architecture specifically designed for efficient deployment of llms in resource-limited settings. loraexit utilizes low-rank adapters to decouple the training of multiple exit interfaces in llms. by enabling the separate optimization of each exit interface, loraexit effectively addresses the performance issues associated with early-exit networks. additionally, we introduce a superior-exit guided distillation method in loraexit, which effectively leverages models of different sizes to aid the training of small models with early exits. in loraexit, the lora adapter enables the training of multiple exits without conflicting gradients. concurrently, our novel batching mechanism facilitates efficient inference. the early exiting strategy further complements this by trimming inference time. the harmonization of these two approaches allows us to optimize both training and inference workflows collectively. to assess the efficacy of loraexit, we conducted extensive experiments in resource-limited settings. our evaluation focuses on real-system speedup as a performance metric, considering that llms commonly operate on modern accelerators like gpus where computation (typically measured in flops) is not the primary bottleneck, but rather factors like memory play a more crucial role @xcite . the experimental results demonstrate that loraexit significantly enhances llm performance, surpassing existing approaches when deployed in such settings. in summary, we make the following contributions: ## methodology in this section, we introduce the proposed adaptive model architecture, which takes advantage of the powerful low-rank adapters to achieve high performance and low overhead (section 3.1). additionally, we introduce a superior-exit guided distillation method, specifically designed for the completely separated exit architecture, which effectively enhances the performance of intermediate exits (section 3.2). finally, we introduce the batched inference operator that can ddresses the computational overhead introduced by employing multiple lora heads (section 3.3). ## experiments and results this section presents the experimental setup, datasets used, and the results obtained through the evaluation of loraexit. ## conclusion in this work, we have introduced loraexit, a novel approach that addresses the limitations of existing early exit networks. by combining a low-rank adapter with the early exit network and implementing a superior-exit guided distillation method, we have demonstrated the effectiveness of loraexit in improving the efficiency of llms during inference. our experimental results showcase a better speedaccuracy tradeoff compared to existing methods, positioning loraexit as a promising solution for deploying llms in resource-limited settings. this advancement opens new avenues for real-time, ondevice language processing applications, offering practical benefits without degrading performance.