P ocket LLM : Enabling On-Device Fine-Tuning for Personalized LLM s (2024)

Abstract: Recent advancements in large language models (LLMs) have indeed showcased their impressive capabilities. On mobile devices, the wealth of valuable, non-public data generated daily holds great promise for locally fine-tuning personalized LLMs, while maintaining privacy through on-device processing. However, the constraints of mobile device resources pose challenges to direct on-device LLM fine-tuning, mainly due to the memory-intensive nature of derivative-based optimization required for saving gradients and optimizer states. To tackle this, we propose employing derivative-free optimization techniques to enable on-device fine-tuning of LLM, even on memory-limited mobile devices. Empirical results demonstrate that the RoBERTa-large model and OPT-1.3B can be fine-tuned locally on the OPPO Reno 6 smartphone using around 4GB and 6.5GB of memory respectively, using derivative-free optimization techniques. This highlights the feasibility of on-device LLM fine-tuning on mobile devices, paving the way for personalized LLMs on resource-constrained devices while safeguarding data privacy.

## introduction the rapidly evolving field of large language models (llms), exemplified by advanced models such as openai's chatgpt, marks a substantial breakthrough in artificial intelligence @xcite . the implications and benefits of the advancements of llms for mobile devices are profound and pervasive. as reported in @xcite , the number of deep models incorporated within individual devices is growing rapidly, making mobile devices are the primary vehicle for ai. the continuous generation of private, inaccessible personal data on mobile devices, often diverging from publicly pre-trained llm distributions, necessitates on-device post-deployment fine-tuning to develop tailored, personalized models while safeguarding data privacy @xcite . on-device fine-tuning of personal data locally is an effective solution for model fine-tuning using personal data while ensuring user data privacy, as all data storage and computation occur exclusively on the device without any data leaving it. fine-tuning current llms on mobile devices with limited resources is challenging due to llms' large size, which demands high computational and memory resources. despite some work claims of achieving on-device fine-tuning using various computation-efficient and memory-saving techniques, these implementations are often demonstrated on edge devices like raspberry pi @xcite rather than on mobile devices such as smartphones and tablets. mobile devices, especially smartphones, more so than other edge devices, generate a substantial amount of highly private and valuable personal data daily due to their extensive usage, holding great potential for enhancing applications by leveraging this data. however, to the best of our knowledge, there have been no successful on-device fine-tuning implementations on mobile devices to date. to bridge this gap, our work aims to enable and optimize the fine-tuning of llms on resource-constrained mobile devices, particularly smartphones. memory is crucial for determining the feasibility of fine-tuning llms on resourceconstrained mobile devices locally, while computational capacity and communication bandwidth primarily impact efficiency, particularly latency. therefore, in this work, our emphasis lies in reducing the memory footprint to make practical finetuning on mobile devices feasible, regardless of efficiency concerns. future efforts are expected to further enhance efficiency. the substantial memory overhead of llm finetuning arises from the computational and storage demands associated with gradients and optimiza-tion states inherent in traditional derivative-based methods. to tackle this challenge on mobile devices, we propose leveraging derivative-free finetuning optimization. this approach aims to reduce the memory footprint during fine-tuning by circumventing the memory-intensive nature of traditional derivative-based methods. our experimental results show that we can fine-tune roberta-large and opt-1.3b on a current off-the-shelf smartphone, oppo reno 6, with a memory consumption of around 4gb and 6.5gb, respectively. we organize our article with the following structure: first, we present related works in section 2, followed by an introduction to our approach (see section 3) and experimental results (see section 4). finally, we conclude with our findings in section 5. moreover, limitations are discussed in section 6. ## related works numerous studies focus on resource-efficient finetuning, which can benefit on-device fine-tuning, categorized into lightweight foundation model design, fine-tuning process optimization, and external resource utilization. moreover, @xcite provides a comprehensive survey on integrating llms with iot devices. ## experiments we conducted experiments using mezo on the oppo reno6 smartphone, which has 12gb of memory. results show mezo can fine-tune roberta-large and opt-1.3b using approximately 4gb and 6.5gb of memory, respectively. in contrast, attempting fine-tuning with adam resulted in an out-of-memory crash. this highlights the memory efficiency of the derivative-free approach, making it viable for fine-tuning llms on resource-constrained devices like smartphones. ## conclusions we demonstrate that derivative-free optimization allows on-device fine-tuning of llms on mobile devices, mitigating the memory constraints of traditional derivative-based methods. experiments show roberta-large and opt-1.3b can be fine-tuned on the oppo reno 6 using 4gb and 6.5gb of memory, respectively. this highlights the advantages of derivative-free optimization for fine-tuning llms on resource-constrained mobile devices. further experiments reveal the efficiency gap between smartphones and gpus, suggesting a need to better utilize hardware capabilities. despite these challenges, our successful implementation of fine-tuning llms on mobile devices is a significant stride towards personalized models while upholding user data privacy.