Distributed LLM Serving on Consumer-Grade GPU s by Reconciling Computation and Communication (2025)

Abstract: Large language models are reshaping internet services. Serving these models is often costly, as it requires multiple high-end GPUs. Consumer-grade GPUs offer cheaper computational power, providing an opportunity for more cost-efficient LLM serving.Prior efforts have explored distributed serving at scale, primarily focusing on model deployment strategies. However, communication efficiency has emerged as a challenge due to the imbalance in data transfer volumes between the two phases of inference: prefill and decode. Prefill requests can involve transmitting up to 1000 times more data than decode requests, leading to decode requests being delayed. Consequently, servers are underutilized while waiting for decode requests. In this paper, we present MoLink, an efficient distributed LLM serving system. It splits the prolonged transmission volume of prefill requests into smaller chunks and carefully scheduling their transmission. It consists of two parts: (i) a transmission scheduling algorithm that fairly determines whether to transmit prefill or decode requests, and (ii) a chunking determination algorithm that determines the transmit volume for prefill requests just-in-time. Our evaluation demonstrates that MoLink reduces TTFT, TPOT, and latency compared to the state-of-the-art distributed LLM serving system, with a maximum reduction of up to 46%.

## introduction large language models represent a groundbreaking shift in generative ai, reshaping existing internet services, ranging from search engines to personal assistants. yet, these advances come with a significant challenge: serving these models can be very expensive. for example, opt-175b requires over 350 gb of accelerator memory for inference. as a result, even basic inference for these llms necessitates multiple high-end gpus within a cluster. in contrast to high-end gpus, consumer-grade gpus offer more affordable computation power. for instance, an rtx 4090 delivers 330 tflops at fp16 precision, compared to the 312 tflops of an a100, with over 4× lower hourly pricing in cloud markets @xcite . it is reported that 101 million pc gpus were shipped in q4 2021 @xcite . although these consumer-grade gpus are widely deployed, they remain underutilized. this presents the opportunity for more cost-efficient llm serving. prior efforts have explored distributed computing at scale. for example, folding@home @xcite sources upwards of 40,000 nvidia and amd gpus. petals @xcite studies the fault-tolerance for serving llms on unsteady servers. hexgen @xcite optimizes the deployment of llms in decentralized environments. helix @xcite discovers optimal llm deployment and request scheduling under heterogeneous clusters. these studies effectively schedule llm deployment for distributed serving. in this paper, we focus on the communication optimization in bandwidth-constrained environments. this is primarily due to the imbalance in transferred data volume between the two phases of inference: prefill and decode. prefill requests can involve transmitting up to 1000 times more data than decode requests, as their volume scales with the number of processed tokens. ideally, many decode requests can be executed concurrently while prefill data is still being transmitted. however, existing systems @xcite @xcite suffer from transmission competition between requests at different phases. for example, the decode request may be delayed by up to one second due to the prefill request occupying the available bandwidth during transmission-even though the decode request itself requires only a few milliseconds to transmit. consequently, the server receiving the decode request remains idle during this period. to address the issue, we present molink, an efficient distributed llm serving system. it mit-igates the transmission competition by splitting the prolonged transmission volume into smaller chunks and carefully scheduling these chunks to transmit. it consists of two part: (i) an transmission scheduling algorithm that determine when to transmit the prefill or the decode requests. it also guarantees that the transmission of prefill requests are not starved. (ii) an chunking determination algorithm that determine the transmit volume for prefill requests just-in-time, so that the decode request is not blocked by the prolonged transmission for prefill request. we summary our contribution as follows: ## evaluation models. we evaluate molink on qwen, a representative and popular open-source transformer model family. specifically, we use qwen 7b @xcite to study the system performance on models. we run model inference with half-precision (fp16). cluster setup. the distributed cluster setup has clusters that contain 3 servers, each server is equipped with a rtx 4090 gpu. inter-node communication has an average bandwidth of 100 mps and an average latency of 30 ms (as our profiling results on lab servers). these servers are configured to use pipeline parallelism for llm serving. we also study the impact of bandwidth, latency and pipeline degree on serving metrics. trace. we use azure conversation [2] to simulate the arrival of requests. it is a representative trace of llm inference invocations with input and output tokens. fig. 7 shows the length distribution of the datasets. fig. 8 shows the arrival rate of the datasets. we remove requests with input lengths larger than 2048 or output lengths larger than 1024, and scale the frequency of requests arrival to fit in the gpus we use, which we indicate as the arrival rate in our experiment. we warm up the cluster for 1 minute and test for 30 minutes. metrics. we measure the latency of the llm service using two key metrics: time to first token (ttft), which indicates the duration of the prefill phase, and time per output token (tpot), which represents the average time taken to generate each token after the first one. additionally, we report the end-to-end latency for each request. these metrics are averaged over the serving duration to provide a comprehensive view of the service's performance. baselines. we compare molink against two state-of-the-art distributed llm serving systems. ## related work machine learning model serving. many recent llm-specific systems tackle the unpredictable execution time and high memory consumption in llm serving. orca @xcite proposed iteration level scheduling to release resources once a request is finished. vllm @xcite introduced pageattention to reduce the memory consumption by allocating the exact number of pages a request requires. speculative inference @xcite applies a small model to predict multiple output tokens. splitwise @xcite and distserve @xcite disaggregates the prompt and decode phase of requests. all the above works are orthogonal to our work. sarathi @xcite introduced chunked prefill, which allocates a budget to the prompt phase. however, it does not optimize the the transmission of prefill requests by chunking the volume. distributed llm serving. several methods explored the potential of utilizing distributed gpus for ml tasks. some of them @xcite co-design the model partition and placement on a heterogeneous cluster. learninghome @xcite and dedloc @xcite studied network-aware routing on a decentralized cluster. swarm @xcite optimize the pipeline communication in a heterogeneous network. there are also several efforts on using approximations to reduce network communication @xcite or synchronization @xcite . skypilot @xcite and mélange @xcite select the best type of gpus for a request. petals @xcite studies a decentralized pipeline parallel setup. it designs a greedy model allocation and request scheduling for a dynamical device group, losing optimization opportunities for a fixed device group. they are only focus on the model placement and request scheduling.