S cale LLM : A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency (2024)

Abstract: Large language models (LLMs) have surged in popularity and are extensively used in commercial applications, where the efficiency of model serving is crucial for the user experience. Most current research focuses on optimizing individual sub-procedures, e.g. local inference and communication, however, there is no comprehensive framework that provides a holistic system view for optimizing LLM serving in an end-to-end manner. In this work, we conduct a detailed analysis to identify major bottlenecks that impact end-to-end latency in LLM serving systems. Our analysis reveals that a comprehensive LLM serving endpoint must address a series of efficiency bottlenecks that extend beyond LLM inference. We then propose ScaleLLM, an optimized system for resource-efficient LLM serving. Our extensive experiments reveal that reveal that with 64 concurrent requests on Mixtral 8x7B, ScaleLLM achieves a 4.3× speed up over vLLM and outperforms state-of-the-arts with 1.5× higher throughput.

## introduction large language models (llms) have significantly changed the field of natural language processing and have been widely used in commercial applications. however, serving llms effectively remains challenging due to system latency, query concurrency, and computational resources constraints. llm applications are typically deployed as online services where users expect real-time responses, while any delay can impact user experience, making low latency to be crucial. also, the computationally intensive nature of llms, which involve inference with billions of parameters, requires substantial computational resources. moreover, achieving scalability to handle multiple concurrent requests without performance degradation further complicates the serving process. latency in llm serving primarily arises from the processing at the serving engine as well as the gateway. the serving engine is the core component responsible for executing the llm inference tasks. it optimizes resource allocation to handle the intensive computational workload of llms to efficiently utilize computational resources, such as gpus. the gateway manages communication between clients (e.g., end-users or applications) and llm instances. it handles incoming requests, directs them to the llm instances, and ensures that responses are returned correctly and efficiently. existing research focuses on optimizing individual subprocedures of llm serving, especially accelerating local inference speeds @xcite ; vllm ai; nvidia). however, in commercial llm applications, end-to-end latency, introduced from functionalities of the gateway, becomes the most significant bottleneck. meanwhile, commercial llm applications have specific requirements on serving, which directly accessing a single llm instance fails to address. in practice, commercial llm applications must satisfy several critical requirements for efficient and reliable inference: i) fault tolerance: there must be replicas of llms to ensure that the serving system can select appropriate replica upon receiving requests under a specific resource constraint, thereby maintaining service reliability even when individual replica instance fails; ii) inference control: the serving system should manage the inference process to ensure that the models are accessed with authentication and can produce responses that are appropriate and safe while adapting to different user demands; iii) low latency: to ensure the user experience, the serving system should process inferences efficiently and deliver responses in real-time; iv) concurrency: small batch sizes and high throughput for individual requests become impractical in real-world llm services such as chatgpt, where the queries can be frequent, e.g., with queries per second (qps) often exceeding 200 @xcite ; v) frugal computational resource usage: given the substantial computational demands, optimizing resource utilization is crucial to prevent excessive costs and ensure the reliable operation of the serving system. thus, a comprehensive llm serving system must balance computational efficiency, concurrency, and latency to manage the high volume of requests. to address the efficiency of llm serving comprehensively, we present scalellm, an optimized llm serving system, as well as an end-to-end measurement, to meet real-world requirements of commercial llm applications. as shown in figure 1 , to address different challenges in commercial llm applications, scalellm optimizes two crucial modules, including i) a routing module that efficiently does replica level load balancing and data transmission; and ii) a strong llm engine to inference promptly with high concurrent requests. our contributions are summarized as follows. ## related work many pre-trained open llms have been released since last year, where the most commonly used models include mixtral 8x7b @xcite and llama-3 @xcite ). such opensource models motivate the industry to build public llm-serving endpoints (together ai; fireworks ai) and empower researchers to work on speeding up the inference speed. flashattention @xcite is proposed to approximate the attention calculation to reduce memory usage with fast computation. by representing the weights and activations with low-precision data types, model quantization @xcite is also widely adopted to reduce memory and computation costs. during llm serving, the key-value cache (kv cache) memory for each request is huge and grows and shrinks dynamically, page attention @xcite is proposed for efficient management of kv cache memory blocks with exact model computation. built on top of pagedattention, vllm (vllm ai) is proposed as a highthroughput distributed llm serving engine that aims to increase gpu utilization and hence speeds up the throughput of llm serving. tensorrt-llm (nvidia) provides industrial-level integration of these state-of-the-art optimization methods with python and c++ runtimes to perform inference efficiently on nvidia gpus. however, these serving engines primarily focus on accelerating local llm computation, neglecting other crucial components such as gateway and routing. to the best of our knowledge, our proposed scalellm is the first to offer an end-to-end latency measurement and optimization specifically for resource-efficient llm serving. ## benchmark llm serving solutions we first provide the end-to-end system breakdown of serving latency in §3.1, then provide the benchmark results of baselines in §3.2. ## optimizations this section discusses the optimization goal, then decomposes the latency into engine latency and gateway latency, and optimizes each component. optimization goal. our goal is to leverage various optimization techniques on both the inference engine and the replica router to improve the end-toend serving performance. the inference engine is applied with different frameworks and optimization methods to increase the throughput and decrease the latency. for the replica router, we break down the latency to engine latency and gateway routing latency. the goal is to decrease the engine latency, especially when the concurrency is high. ## blueprint architecture of dynamic inference load balancing system our experiments have revealed that different engine parameters are suited for different throughput loads, thereby emphasizing the need for a dynamic load balancing system for ai inference unifying the strengths of these heterogeneous configurations and averaging out weakness. we propose a blueprint for such a dynamic inference load balancing system, designed to optimize resource allocation by efficiently distributing inference requests across these heterogeneous replicas, thereby maintaining consistently high throughput regardless of the concurrency scale. the core component of the proposed system is a dynamic inference balancing router that handles incoming inference requests and intelligently routes them to the appropriate replica based on a routing policy, mapping request concurrency levels to throughput ranges and selecting the replica best suited to manage the specific workload range. the dynamic routing policy illustrated in figure 8 , showcasing the blueprint architecture and the policy breakdown follows a general rule of thumb: low concurrency (< 64 requests). route requests to nodes with fewer replicas but higher tensor parallelism to optimize resource utilization for smaller batch computations. high concurrency (≥ 64 requests). route requests to nodes with more replicas but lower tensor parallelism, effectively distributing the workload to squeeze everything out of available compute by leveraging the power of replica parallelism. ## conclusion and future work in this paper, the proposed scalellm framework optimizes both the llm serving engine and the platform. as llm applications grow in complexity, platform latency becomes increasingly critical. instead of focusing solely on local inference speed, industrial research should prioritize reducing end-to-end latency by streamlining the serving gateway and optimizing the platform-level performance. llm can also be deployed in a federated way to further reduce the latency @xcite . from the user's point of view, the metrics to measure an inference system are close to their intuitive feeling, which is the following four metrics: • throughput: the number of output tokens per second that an inference server can generate across all users and requests.