Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference (2024)

Abstract: The customization of large language models (LLMs) for user-specified tasks gets important. However, maintaining all the customized LLMs on cloud servers incurs substantial memory and computational overheads, and uploading user data can also lead to privacy concerns. On-device LLMs can offer a promising solution by mitigating these issues. Yet, the performance of on-device LLMs is inherently constrained by the limitations of small-scaled models. To overcome these restrictions, we first propose Crayon, a novel approach for on-device LLM customization. Crayon begins by constructing a pool of diverse base adapters, and then we instantly blend them into a customized adapter without extra training. In addition, we develop a device-server hybrid inference strategy, which deftly allocates more demanding queries or non-customized tasks to a larger, more capable LLM on a server. This ensures optimal performance without sacrificing the benefits of on-device customization. We carefully craft a novel benchmark from multiple question-answer datasets, and show the efficacy of our method in the LLM customization.

## introduction large language model (llm) has achieved unprecedented success on diverse natural language processing tasks such as machine translation, question and answering, text summarization and stylization, etc. now then, it is expected for llmpowered artificial intelligence (ai) to understand and satisfy each user's unique needs such as recommendation systems, personalized assistance, and personalized search. to this end, a pivotal cornerstone is customized llm where the llm is highly advanced to a user-requested task. indeed, several web services related to llm customization have been emerged such as gpts @xcite and personaai @xcite . however, due to the significant scale of llm, keeping all the customized llms in the servers imposes a tremendous burden. also, the privacy issues is inevitably raised by uploading the user's data which entail user-requested task. then, the focus is shifting towards on-device llm. however, as the limited computing power of edge devices, it is impractical to address models as large as those on the servers. therefore, for on-device llm customization, it is crucial to maintain the performance on user-defined tasks, while constraining the model sizes. the practical method for the on-device llm customization, however, has been less explored. recently, several works are developed to further lead out the ability of llms, and they may be exploited to cover the performance limit of llms on smaller size (device-level). @xcite introduced few-shot learning where a few example query-answer prompts are given together with users' queries. chain-of-thought (cot) @xcite tried in-context learning by encouraging llm to generate evidences as well as final answers. also, for knowledge-intensive nlp tasks, retrieval augmented generation (rag) @xcite made up query-relevant prompts by retrieving a given database. these prompt-based approaches have a intrinsic problem of increasing inference cost as the prompts get long and complex, and hence they are not suitable for edge devices. moreover, we can consider fine-tuning the ondevice llm in order to internalize these promptbased knowledge for user-specific tasks to the model. freezing the pre-trained llm, adapterbased methods @xcite @xcite have tried to facilitate llm fine-tuning, and the low-rank-based adapter lora @xcite have been most in the limelight. despite on-device-scale llm armed with the generation of customized adapter sec. 3.2 sec. 3.3 medical adapter server llm if not confident adapters, fine-tuning is time-consuming process and also needs a certain-level of training dataset. however, computing power of edge devices is limited and collecting enough user-specific data is also impractical. thus, we raise inquiry: how about simply customizing llm without on-device training? for this purpose, given a target customization task, we propose crayon customizing the ondevice llm via a single customized adapter which is blended on-the-fly from a set of base adapters, called adapter pool. to cover a wide range of user requests, the base adapters are learned to contain different knowledge each other. as depicted in fig. 1 , our approach requires no additional training cost in both of server and edge device when blending the customized adapter. in addition, we develop a device-server hybrid inference strategy to effectively leverage the better-generalized larger model of the server for handling unexpected queries (out-of-scope for the customized model). our contributions are summarized as follows: defining & processing customized task. in the context of few-shot learning @xcite , an llm is prompted by several query-answer pairs to better understand testing queries. it has been proven that these few-shot prompts are helpful for increasing the generalization capability even in smaller llms. however, prompting increases the inference cost of llms as well, which is not preferred to on-device use case. rather than prompting the few-shot examples d c , we define it as a target task specified to the user. from them, we immediately generate an adapter customized for the target task on the server, and deploy it to the on-device llm. note that, considering the privacy issue and communication cost with the server, d c itself is never transmitted to the sever in our method. baseline llm. for autoregressive, causal language model, most of popular llms such as gpt @xcite , llama @xcite , and falcon @xcite ) have adopted the decoderonly transformer @xcite architecture. hence, in our work, we select the smallest llama (llama-7b) as the baseline on-device llm, which is reasonable size for edge devices foot_0 . adapter for customized llm. to reduce training cost, parameter efficient fine-tuning (peft) injects small trainable adapters, and only updates them for llm fine-tuning. as a widely-used peft approach, lora @xcite approximates the gradient of pre-trained weights into low-rank matrices, and use them as the llm adapters. as such, we also employ lora as our llm adapter. note that the learning lora does not take place on edge devices in our method. briefly explaining our approach, we only train a set of n base loras {l θn } n @xmath0.e., lora pool) given a training set d tr on the server, and then they are combined and deployed for instant llm customization to the target task (i.e., d c ) without additional training. device-server hybrid inference. although an ondevice llm is well-customized to a user-specified task, there is inevitable performance gap between the device-level and server-level llms. especially, the on-device llm suffers from more performance drop when the inputs are out of the target task. hence, we devise a device-server hybrid inference strategy. when output of an on-device llm is unconfident, the output is replaced from the server's larger model. to reduce frequent use of the server llm, we develop a method to determine the reliability of on-device llm's output inside the device. ## methodology in this section, we introduce crayon which consists of lora pool construction (sec. 3.2) and customized lora generation (sec. 3.3). also, we develop device-server consistent inference (sec. 3.4). ## conclusions we propose crayon and device-server hybrid inference for customizing on-device llm for the first time. in crayon, we jointly train a pool of base loras, which has distinct knowledge and characteristics each other. using this base lora pool, we can instantly blend the base loras into a customized lora for user-defined task, without additional training or transferring user data to the server. to encompass complex queries, we develop the device-server hybrid system that measures the reliability of the customized llm to identify the necessity of server llm. we also present a new benchmark for quantitative evaluation of on-device llm customization, incorporating commonly-used qa datasets, and our method shows superior performance. we hope that this benchmark can be a valuable tool for future research in this field.