L ingua L inked: Distributed Large Language Model Inference on Mobile Devices (2024)

Abstract: Deploying Large Language Models (LLMs) locally on mobile devices presents a significant challenge due to their extensive memory requirements. In this paper, we introduce LinguaLinked, a system for decentralized, distributed LLM inference on mobile devices. LinguaLinked enables collaborative execution of the inference task across multiple trusted devices and ensures data privacy by processing information locally. LinguaLinked uses three key strategies. First, an optimized model assignment technique segments LLMs and uses linear optimization to align segments with each device's capabilities. Second, an optimized data transmission mechanism ensures efficient and structured data flow between model segments while also maintaining the integrity of the original model structure. Finally, LinguaLinked incorporates a runtime load balancer that actively monitors and redistributes tasks among mobile devices to prevent bottlenecks, enhancing the system's overall efficiency and responsiveness. We demonstrate that LinguaLinked facilitates efficient LLM inference while maintaining consistent throughput and minimal latency through extensive testing across various mobile devices, from high-end to low-end Android devices.

## introduction the past decade has witnessed a seismic shift in the machine learning (ml) landscape, particularly with the rise of large language models (llms), which are built atop transformer decoders @xcite . these llms @xcite @xcite @xcite @xcite have achieved stateof-art performance on natural language processing (nlp) benchmarks such as text generation, question answering, machine translation, and text 0 * equally contributed. summarization, and led to commercial offerings such as openai chatgpt and github copilot. recent research has established that as the number of parameters in these models increases, they demonstrate enhanced capabilities in various language tasks @xcite @xcite @xcite . however, deploying these llms on mobile devices is challenging due to their significant memory and processing requirements. traditional server-based inference raises privacy and bandwidth issues. an alternative is distributed inference, where llms are split into smaller segments across multiple devices, reducing the need for heavy model weight quantization and maintaining accuracy. while previous studies have looked into distributed model deployment on mobile computing platforms @xcite @xcite , these have largely concentrated on smaller-scale models used in computer vision applications, which have a much smaller memory footprint compared to llms and 1 160 typically do not need iterative inference. in this paper, we present lingualinked foot_0 , a decentralized distributed inference system for llm deployment on mobile devices. the core concept behind lingualinked is distributing model segments across 'trusted' devices shown in figure 1 , such as personal smartphones and tablets. this approach overcomes the limitations of individual device capacities, privacy concerns, and bandwidth constraints. however, it faces challenges like managing diverse device capabilities, handling data dependencies between model segments, and adjusting to dynamic resource availability. lingualinked addresses these challenges with three key components: optimized model assignment that aligns model segments with device capabilities while minimizing data transmission, runtime load balancing to redistribute tasks and prevent bottlenecks, and optimized communication to ensure efficient data exchange between model segments. we perform a thorough evaluation of lingualinked on high-end and low-end android devices. in a single-threaded setting, compared to the baseline, lingualinked achieves an inference performance acceleration of approximately 1.11× to 1.61× across both quantized and fullprecision models. with multi-threading, the system exhibits further improvements, achieving acceleration rates of approximately 1.73× to 2.65× for both quantized and full-precision models. runtime load balancing yields an overall inference acceleration of 1.29× to 1.32×. importantly, our findings indicate that lingualinked's performance gains are more pronounced with larger models, suggesting enhanced scalability and effectiveness in handling complex, resource-intensive tasks. we develop an android application to demonstrate lin-gualinked's effectiveness in a typical mobile computing environment, showing how llms can operate on devices with diverse capabilities. ## lingualinked as shown in figure 2 , the lingualinked system facilitates llm distribution across mobile devices by transforming the llm into a computational graph on a coordinator server, then partitioning it into sub-modules for optimized allocation to devices based on their performance metrics. it employs primary and secondary optimizers for task distribution and load balancing, with a communication strategy that minimizes data transmission between devices, ensuring efficiency and privacy as all data remains local to the devices. ## discussion a major direction for expanding lingualinked involves adapting it for distributed fine-tuning on mobile devices, allowing model customization based on user interactions and local data, paving the way for personalized ai applications while preserving data privacy. we also envision extending lingualinked to handle multi-modality models, enhancing its applicability in diverse real-world scenarios. to further improve lingualinked, we envision more advanced model computational graph partitioning strategies involving further optimizations on task divisions better aligned with device capabilities. moreover, integrating advanced load balancing algorithms that account for not only computational capabilities but also battery life and user engagement patterns will ensure a holistic approach to distributed computing on mobile platforms. finally, the implications of this research are significant for ai policy, as it challenges the prevailing reliance on cloud infrastructure and centralized data centers for ai deployment. by demonstrating the potential to deploy ai systems from a network of mobile devices, our work suggests a paradigm where the 'means of production' for ai can be decentralized and localized. this model of deploy-ment could lead to a future where ai systems are both operated and fine-tuned locally using a diverse array of small devices. such a setup could make ai systems more difficult to regulate, as the distribution and localization of ai technologies allow for widespread, generic hardware use. ## conclusion in this work, we introduce lingualinked, a system for decentralized llm inference on mobile devices. to the best of our knowledge, lingualinked is the first work that exploits deploying llm distributively on mobile devices. lingualinked implemented optimized model assignment strategy, network communication and runtime load balancing mechanism to accelerate the distributed llm inference on mobile devices. this approach tackles the complexities of deploying both full precision and quantized llms of various sizes within mobile computing environments.