D i SC o: Device-Server Collaborative LLM -based Text Streaming Services (2025)

Abstract: The rapid rise of large language models (LLMs) in text streaming services has introduced significant cost and Quality of Experience (QoE) challenges in serving millions of daily requests, especially in meeting Time-To-First-Token (TTFT) and Time-Between-Token (TBT) requirements for real-time interactions. Our real-world measurements show that both server-based and on-device deployments struggle to meet diverse QoE demands: server deployments face high costs and last-hop issues (e.g., Internet latency and dynamics), while on-device LLM inference is constrained by resources. We introduce , a device-server cooperative scheduler designed to optimize users’ QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. employs cost-aware scheduling, leveraging the predictable speed of on-device LLM inference with the flexible capacity of server-based inference to dispatch requests on the fly, while introducing a token-level migration mechanism to ensure consistent token delivery during migration. Evaluations on real-world workloads—including commercial services like OpenAI GPT and DeepSeek, and open-source deployments such as LLaMA3—show that can improve users’ QoE by reducing tail TTFT (11-52%) and mean TTFT (6-78%) across different model-device configurations, while dramatically reducing serving costs by up to 84% through its migration mechanism while maintaining comparable QoE levels.

## introduction large language models (llms) have revolutionized various applications, with over 60% focusing on conversational interactions such as chatbots @xcite . meeting high serving demands requires scaling deployments across * corresponding author on-premise servers in the cloud and on-device inference, as seen in apple intelligence @xcite ) and google's gemini nano @xcite . the quality of experience (qoe) for interactive applications is primarily evaluated by two critical metrics: time-to-first-token (ttft) in the prefill stage, which quantifies the initial response latency, and time-between-token (tbt) during the decode stage, which measures the consistency of token delivery speed @xcite . on-server deployments lower serving costs by sharing infrastructure among many requests but often introduce unpredictable high latency due to request queuing delays @xcite and the internet speed fluctuations. while on-device deployment is able to serve increasingly capable llms with sufficient accuracy, it suffers from slow processing speeds for long prompts and high energy consumption. for example, a fully-charged iphone can only operate for less than two hours running an llm with 7b parameters @xcite . this paper introduces a novel paradigm for costconstrained device-server cooperative inference. we incorporate both server usage (e.g., monetary costs) and device energy costs via a dynamic exchange rate, which can be adjusted by endpoint users to balance response generation between the cloud and devices. as such, we can strategically distribute inference requests between endpoints and dynamically migrate ongoing token generation to maximize qoe. however, realizing this vision presents several fundamental challenges: as shown in figure 1 , we introduce disco, a device-server cooperative scheduler that addresses these challenges via two key innovations: through extensive evaluation using real-world traces from commercial llm streaming api services, including gpt and deepseek, and on-device deployments, we demonstrate that disco improves mean and tail ttft by up to 50% without tbt violation, significantly reducing costs. overall, we make the following contributions: ## characterizing llm inference this section characterizes the llm inference performance of on-server and on-device paradigms, which informs our design. we evaluate four commercial streaming llm apis: openai's gpt-4o-mini (openai, 2024), deepseek's deepseek-v2.5 (deepseek, 2024), cohere's command @xcite , and hyperbolic-hosted llama-3-70b-instruct (hyperbolic, 2024). for on-device analysis, we deploy qwen-2.5-7b-instruct @xcite and llama-3.1-8b-instruct (grattafiori et al., 2024) on both server-grade (nvidia a40, 48gb) and consumer-grade (dual nvidia rtx 3080, denoted as 3080x2) gpus. we sample 1,000 requests from the alpaca dataset @xcite , following a poisson distribution with a mean request arrival interval of 30 seconds. ttft characteristics. our measurements reveal the contrasting ttft patterns between on-device and on-server inference. as shown in figure 2 , on-device inference exhibits stable ttfts when processing identical prompts at 60-second intervals, primarily reflecting the prefill duration due model deployment pearson coef. to dedicated local hardware resources. in contrast, on-server inference experiences high variations and significant tail latency, attributed to network delays, request queuing, and resource contention. unlike previous works that focus solely on prefill latency (e.g., @xcite ) or the sum of queuing and prefill (e.g., @xcite ), we measure user-perceived ttft as a comprehensive sum of network, queuing, and prefill latencies. our extensive evaluations across four major llm providers consistently demonstrate this improvement. this methodology aligns with industry benchmarks, such as artificial analysis (artificial analysis, 2025)'s continuous monitoring of diverse llm services. notably, since most on-device llms struggle with prefilling or understanding long prompts, we focus on short prompts, where request queuing delay and batch completion waiting time during generation dominate the overall ttft. we summarize the ttft performance of 1,000 requests in tbt characteristics. tbt characterizes the i/obound decode stage latency. analysis of temporal samples and distributions across six setups (figure 3 ) reveals higher tbt variability in on-server inference compared to on-device execution. moreover, both deployments achieve generation speeds exceeding user consumption rates ( § 2.2), making cooperative serving practical. opportunities and challenges. our studies further reveal that as on-device models continue to improve-often fine-tuned for specific tasks @xcite )they are progressively achieving performance parity with on-server models in popular applications like instruction-following and translation (detailed in §5 and appendix d). however, deploying these models on-device introduces challenges such as long prefill latency and startup overhead. on the other hand, our real-world studies of conversational workloads highlight key opportunities: (i) on-server ttft is largely unpredictable and shows minimal correlation with prompt length, whereas on-device ttft scales nearly linearly with prompt length and is highly predictable; and (ii) both paradigms achieve token generation speeds that exceed typical user consumption rates. by integrating these findings-especially the predictable performance of on-device inference and the elastic scaling capabilities of server-based inference-we observe opportunities for optimization in cost-constrained device-server cooperative serving. dynamic request migration between server and device endpoints during response generation can yield significant cost savings. ## disco policies disco optimizes both qoe and cost through (1) dispatch control that determines where to initiate token generation, and (2) migration control that enables dynamic handoff during generation. the dispatch controller optimizes ttft by strategically routing requests, while the migration controller maintains consistent tbt while reducing costs. ## evaluation through extensive experimentation with four production-grade llm services and state-of-theart open-source models, we demonstrate disco's exceptional performance. our rigorous evaluation spanning diverse deployment scenarios reveals that disco delivers remarkable improvements, reducing both mean ttft (6-78%) and tail ttft (11-52%) while achieving cost savings of up to 83.6%. ## conclusions this paper introduces disco, a device-server cooperative scheduler that addresses qoe and cost challenges in llm serving for real-time conversational applications for end users. disco uses cost-aware scheduling and token-level migration to dynamically optimize ttft and tbt across device and server endpoints. our evaluations on real-world traces from platforms like gpt and deepseek show that disco significantly improves both ttft and tbt while reducing costs.