Xinference: Making Large Model Serving Easy (2024)

Abstract: The proliferation of open-source large models necessitates dedicated tools for deployment and accessibility. To mitigate the complexities of model serving, we develop Xinference, an open-source library designed to simplify the deployment and management of large models. Xinference effectively simplifies deployment complexities for users by (a) preventing users from writing code and providing built-in support for various models and OpenAI-compatible APIs; (b) enabling full model serving lifecycle management; (c) guaranteeing efficient and scalable inference and achieving high throughput and low latency. In comparative experiments with similar products like BentoML and Ray Serve, Xinference outperforms these tools and offers superior ease of use.Xinference is available at https://github.com/xorbitsai/inference.

## introduction recently, open-source large models are quickly catching up with the closed-source models @xcite @xcite . there is a growing demand to deploy these open-source models in private user environments, as an increasing number of ai applications and even non-ml/ai practitioners require a straightforward and effective inference toolkit for managing model deployment. although there are inference engines and frameworks for large model inference @xcite , many current inference toolkits are not as simple and convenient to use. therefore, this paper focuses on how to build an efficient and easy-to-use inference toolkit. streamlining large model inference is crucial. open-source large models, customizable and free from data privacy concerns, are highly suitable for private deployment. a simple toolkit can enable * corresponding authors: xuye qin and yueguo chen. more users to access the capabilities of large models and focus on ai applications rather than spend time managing inference services. however, building an easy-to-use inference toolkit is a non-trivial task. first, ai applications often rely on different types of models, such as chat, embedding, or multimodal models, along with technologies like function calling @xcite or retrieval-augmented generation (rag) @xcite . each model type or technology mentioned possesses distinct characteristics and may require specific configurations. second, the landscape of inference engines and hardware is vast, with options like vllm @xcite , llama.cpp @xcite , sglang @xcite , and various cpus and gpus, such as x86, apple silicon, nvidia, amd. a particular inference engine is typically designed for specific users and application scenarios. third, users need to scale inference workloads onto clusters to achieve high throughput and low latency. therefore, there is a need for a framework-agnostic inference toolkit to manage multiple models and various inference engines while providing users with convenient and user-friendly services. several toolkits, such as ray serve @xcite and bentoml @xcite , aim to facilitate serving services for deploying various models across diverse hardware platforms. however, we have found that these tools fail in user-friendliness, often requiring extensive coding, or suffer performance degradation. for example, first, bentoml and ray serve both require users to write code to deploy models, which can be quite challenging for users who are not familiar with model inference. second, these two tools do not cover the full model serving lifecycle and lack essential features. third, bentoml suffers performance degradation when scaling models replicas. to address the aforementioned issues, we have developed xinference, an inference toolkit de-signed to streamline the serving of large models. first, it is designed for ease of use, eliminating the need for users to write additional code, and provides built-in support for various of models, features, and inference engines. second, it can manage the entire lifecycle of model serving, from scaling models to clusters to managing computing resources. third, it has minimal performance loss when integrating with an inference engine, ensuring high throughput and low latency on clusters, and offers scheduling optimizations. xinference leverages xoscar @xcite , an actor programming framework we designed, as its core component to manage machines, devices, and inference engines. each actor serves as a basic unit for model inference tasks and different inference engines can be integrated into the actor, enabling us to manage multiple inference engines and model replicas. xinference targets a broad range of audiences, including ai application developers, ml engineers, and even non-ai/ml practitioners who simply wish to use large ai models. it is opensourced with the apache 2.0 license and available on github 1 . experiments demonstrate that, compared to ben-toml and ray serve, xinference maintains excellent latency and throughput across various workload scenarios. when deploying a single model replica, compared to the original inference engine, xinference's performance loss is within 3.64%. ## background and motivation in this section, we outline the motivation and the design principles for the user-centric inference service. ## system implementation in this section, we show how xinference manages models and supports scalable inference. ## conclusion in conclusion, xinference is a user-friendly tool designed for large model serving. this tool eliminates the need for users to write additional code or configurations, allowing users to focus on building ai applications. it can manage the entire lifecycle of large model serving. xinference can scale serving workloads onto a cluster and achieve high throughput and low latency. at its foundation, xinference employs the actor framework that we developed to handle the management of inference engines and hardware.