HP ipe: Large Language Model Pipeline Parallelism for Long Context on Heterogeneous Cost-effective Devices (2024)

Abstract: Micro-enterprises and individual developers emerge analysis demands for long sequence with powerful Large Language Models (LLMs). They try to deploy the LLMs at local, but only possess various commodity devices and the unreliable interconnection between devices. Existing parallel techniques do not lead to the same effectiveness in limited environment. The heterogeneity of devices, coupled with their limited capacity and expensive communication, brings challenges to private deployment for maximized utilization of available devices while masking latency. Hence, we introduce HPipe, a pipeline inference framework that successfully mitigates LLMs from high-performance clusters to heterogeneous commodity devices. By ensuring a balanced distribution of workloads, HPipe facilitates the parallel execution of LLMs through pipelining the sequences on the token dimension. The evaluation conducted on LLaMA-7B and GPT3-2B demonstrates that HPipe holds the potential for context analysis on LLM with heterogeneity devices, achieving an impressive speedup in latency and throughput up to 2.28 times.

## introduction the emergence of llms has significantly enhanced automated content comprehension, as they adeptly capture semantic information within extensive contexts. enterprises employ techniques such as sentiment analysis @xcite @xcite and content analysis @xcite to harness the potential value to facilitate the anticipation of user engagement and strategic decision-making. however, due to the stringent memory and computational requirements of llms, they are commonly deployed on highperformance computing clusters. the advanced devices and high-velocity transmission like nvlink, boasting transfer rates approaching 900 gb/s, enable rapid computation and efficient synchronization. while micro-enterprises introduce demands to leverage the private llm, they only have inconsistent weaker devices. the interconnection among these devices also suffers from limited bandwidth. devices connected via wireless network exhibits transfer rate merely up to 1 gb/s. thus, the customized llm deployment schema for microenterprises deserves further exploration. for the demands of effective inference, inference engines @xcite provides hybrid data and pipeline parallelism @xcite and combined with tensor parallelism @xcite . in high-performance computing centers, they substantially alleviate computational and memory pressure, thereby augmenting inference speed and enhancing throughput. however, existing methods cannot be directly applicable to the scenarios of micro-enterprises. the deployment for the micro-enterprises presents several problems. 1) extended text: as llm support longer inputs, the expanded context window brings higher arithmetic pressure. the micro-batch pipeline struggles to maintain efficiency. each stage of the pipeline demands longer processing durations, and the coarser granularity diminishes the parallelism. 2) communication discrepancy: the conditions for communication between devices are discrepant. gpus within a device generally exchange data via pcie, and gpus between devices rely on the network. this impedes the efficacy of communication-intensive methods such as tensor parallelism. 3) heterogeneous devices: it is essential that integrating heterogeneous devices to employ all available resources for micro-enterprises. the dual heterogeneity of both computation and transmission, coupled with expensive communication, bring challenges to orchestrating the available devices of micro-enterprises for llms deployment. to address these challenges, we propose hpipe, a pipeline inference framework dedicated to content comprehension for private llms. it deploys the llms on heterogeneous devices with pipeline parallelism on the token dimension. hpipe shields the heterogeneity of devices by distributing llms based on computing capabilities and transmission conditions. for extended context, hpipe slices them into segments by a dynamic programming algorithm and pipelines the computation of segments to amplify the degree of parallelism. hpipe successfully mitigates llms from high performance clusters to heterogeneous devices, achieving up to a 2.28× increase in both latency and throughput, alongside a 68.2% reduction in energy consumption compared to other methods. 2 background and motivation 2.1 parallelism pipeline and tensor parallelism are two popular methods for accelerating the inference of llms as shown in fig. 1. matrix multiplication (mat-mul) contributes to most of the overall computation amount. solving a matmul can be converted into the solving sum of several smaller matmul. tensor parallelism leverages this by dividing and distributing the weight matrix to multiple devices to enable the computation in parallel. once the computation completes, devices will communicate to synchronize the results. thus, tensor parallelism is commonly used when the transmission is guaranteed. the pipeline mechanism distributes llms across multiple devices, with each device dedicated to a stage of computation. the request is usually segmented into micro-batches and processed sequentially. transmission is only required for intermediate result. while pipeline is communication lightweight, pipeline in batch dimension still bring challenge when llms are serving for microenterprises. memory constraints limit the batch size of requests, which reduces space of dividing data and hinders the degree of parallelism. moreover, as sequence length increases, each pipeline stage spends more time. the increasing execution time of stages introduces more idle waiting. ## related work parallel acceleration on deep neural networks has been widely studied. only using the data parallelism @xcite @xcite is not enough as parameters of llms expand. pipeline parallelism @xcite ; aminabadi @xcite and tensor parallelism @xcite distribute the model to multiple gpus, thus reducing the memory burden of the device and allowing efficient scaling of llm inference. on the basis of them, lots of work achieve inference speedup. byte-transformer @xcite proposes a padding-free algorithm that liberates inference from redundant computations on zero padded tokens when faced with variable-length sequences. kernel fusion @xcite optimized cuda kernels to reduce memory access and improve computation speed. these methods focus on latency-oriented scenarios with advanced devices, limiting their deployment to easily accessible hardware with weaker computing capability and memory storage. in comparison, this paper derives the parallelism schema on a heterogeneous cluster of commodity devices to cater to the private application requirements. in addition, techniques proposed by hpipe are orthogonal to the optimized methods, including quantization @xcite and kernel optimization @xcite , hence they can be combined with them for better performance. ## conclusion this paper introduces hpipe, an inference framework to accelerate the content analysis with llms prototyped on the cluster of commodity devices. it effectively integrates computing resources, allowing a fine-granularity pipeline on heterogeneous devices. hpipe demonstrates the potential to accelerate llms inference with long sequence input, offering a solution for llms deployment in hetero-