Taming the Titans: A Survey of Efficient LLM Inference Serving (2025)

Abstract: Large Language Models (LLMs) for Generative AI have achieved remarkable progress, evolving into sophisticated and versatile tools widely adopted across various domains and applications. However, the substantial memory overhead caused by their vast number of parameters, combined with the high computational demands of the attention mechanism, poses significant challenges in achieving low latency and high throughput for LLM inference services. Recent advancements, driven by groundbreaking research, have significantly accelerated progress in this field. This paper provides a comprehensive survey of these methods, covering fundamental instance-level approaches, in-depth cluster-level strategies, and emerging scenarios. At the instance level, we review model placement, request scheduling, decoding length prediction, storage management, and the disaggregation paradigm. At the cluster level, we explore GPU cluster deployment, multi-instance load balancing, and cloud service solutions. Additionally, we discuss specific tasks, modules, and auxiliary methods in emerging scenarios. Finally, we outline potential research directions to further advance the field of LLM inference serving.

## introduction with the rapid evolution of open-source large language models (llms), weekly updates to model architectures and capabilities have become the norm in recent years. the surging demand for these models is evident from huggingface download statistics, which range from hundreds of thousands for models like @xcite @xcite , and llama-3.3-70b-instruct @xcite to millions for deepseek-v3 @xcite and @xcite over recent months. however, when deploying these * corresponding author models, their large-scale parameters and attention mechanisms impose substantial demands on memory and computational resources, presenting significant obstacles to achieving the desired low latency and high throughput in processing requests. these challenges have spurred extensive research across multiple domains of inference serving optimization to meet service level objectives (slos). this paper presents a systematic survey of llm inference serving methods, organized hierarchically from instance-level optimizations to clusterscale strategies and emerging scenarios. instance-level optimization ( §3) begins with model placement ( §3.1), essential for distributing parameters across devices when single-gpu memory is insufficient. subsequent request scheduling ( §3.2) prioritizes batched processing through decoding length prediction ( §3.3), where shorter requests are prioritized to reduce overall latency. dynamic batch management then governs request insertion/eviction during iterative processing. while kv cache ( §3.4) mitigates redundant computation, challenges persist in storage efficiency, reuse strategies, and compression. due to the distinction between the prefill and decoding phases, the disaggregated architecture ( §3.5) was introduced, facilitating the optimization of each phase. and how multiple llms efficiently share the same computing resources through multiplexing ( §3.6). cluster-level optimization focuses on deployment strategies ( §4), particularly cost-effective gpu cluster configurations with heterogeneous hardware, as well as service-oriented cluster scheduling ( §4.1). scalability introduces load balancing challenges ( §4.2) to prevent resource underutilization or overload across distributed instances. when local hardware infrastructure is inadequate to fulfill deployment requirements, cloud-based solutions ( §4.3) are necessary to address dynamic llm serving demands. ## background this section provides an overview of llm fundamentals, aimed at enhancing the understanding of inference serving, along with the relevant evaluation metrics. ## future works given the rapid evolution of llm inference services, we present several recommendations for future research. other directions, though smaller, are significant in impact, are presented at §a. we hope that these suggestions will provide valuable insights for advancing future research. ## conclusion the primary challenge in llm inference serving stems from the significant memory requirements caused by the scale of parameters and the computational load associated with attention mechanisms. this paper presents a thorough and hierarchical review of methods, encompassing approaches from basic instance-level to more advanced cluster-level techniques, as well as a variety of emerging scenarios. additionally, we explore small yet significant areas and suggest potential directions for future research. we hope this work provides valuable insights for ongoing research in this crucial field.