Petals: Collaborative Inference and Fine-tuning of Large Models (2023)

Abstract: Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈1 step per second, which is enough for many interactive LLM applications. Unlike most inference APIs, Petals also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods. The system, its source code, and documentation are available at https://petals.mlVideo (2 min): https://youtu.be/F4muLI-0hTE

## introduction in recent years, the nlp community has found that pretrained language models can solve many practical tasks, through either fine-tuning @xcite or simple prompting @xcite . furthermore, performance tends to improve as scale increases @xcite . following this trend, modern llms often have hundreds of billions of parameters @xcite @xcite . some of these llms were released publicly @xcite @xcite . most recently, the bigscience project has released bloom, a 176 billion parameter model supporting 46 natural and 13 programming languages @xcite . while the public availability of 100b+ parameter models makes them easier to access, they remain difficult to use for the majority of researchers and practitioners due to memory and computational costs. for instance, opt-175b and bloom-176b need over 350 gb accelerator memory for inference and significantly more for fine-tuning. as a result, these llms usually require multiple highend gpus or multi-node clusters to be run. both of these options are extremely expensive, which limits research and potential applications of llms. several recent works aim to democratize llms by "offloading" model parameters to slower but cheaper memory (ram or ssd), then running them on the accelerator layer by layer @xcite . this method allows running llms with a single low-end accelerator by loading parameters from ram justin-time for each forward pass. offloading can be efficient for processing many tokens in parallel, but it has inherently high latency: for example, generating one token at a time with bloom-176b takes at least 5.5 seconds for the fastest ram offloading setup and 22 seconds for the fastest ssd offloading. in addition, many computers do not have enough ram to offload 175b parameters. another way to make llms more accessible is through public inference apis, where one party hosts the model and lets others query it over the internet (openai; ai21; forefront). since most of the engineering work is done by the api owner, this is a relatively user-friendly option. however, apis are often not flexible enough for research use: there is no way to change the model control flow or access internal states. on top of that, current api pricing can make some research projects prohibitively expensive @xcite . in this work, we explore an alternative strategy inspired by crowdsourced distributed training of neural networks from scratch (ryabinin and gusev, 2020). we introduce petals, a platform that allows multiple users to collaborate and perform inference and fine-tuning of large language models over the internet. each participant runs a server, a client or both. a server hosts a subset of model layers (typically, transformer blocks) and handles requests from clients. a client can form a chain of pipeline-parallel consecutive servers to run the inference of the entire model (section 2.1). aside from inference, participants can fine-tune the model through parameter-efficient training methods like adapters @xcite or prompt tuning @xcite or by training entire layers (section 2.2). once trained, submodules can be shared on a model hub (section 2.3), where others can use them for inference or further training. we demonstrate that existing 100b+ models can run efficiently in this setting with the help of several optimizations: dynamic quantization, prioritizing low-latency connections, and load balancing between servers (section 3). finally, we discuss limitations and possible future work (appendix a). ## design and use cases practical usage of large language models can be broadly divided into two main scenarios: inference and parameter-efficient adaptation to downstream tasks. in this section, we outline the design of petals, showing how it handles both scenarios and also allows easily sharing trained adapters between the users of the system. ## internal structure and optimizations one of the primary considerations for distributed inference is its performance. it can be broken down into three main aspects: computation speed (5-yearold gaming gpu vs. new data center gpu), communication delay due to distance between nodes (intercontinental vs. local), and communication delay due to bandwidth (10 mbit/s vs. 10 gbit/s). in terms of raw flops, even consumer-grade gpus like geforce rtx 3070 could run a complete inference step of bloom-176b in less than a second (nvidia, 2020). however, the gpu memory can only hold a small fraction of model layers: running naïvely would require 44 rtx 3070 gpus and 44 communication rounds. to make this more efficient, we use quantization to store more parameters per gpu, reducing the number of consecutive devices and communication rounds (section 3.1). on top of that, each client prioritizes nearby servers to make communication rounds faster (section 3.2). ## conclusion this paper introduces petals, a system for efficient collaborative inference and fine-tuning of large language models. we offer a user-friendly generation interface and a flexible api to access models served over the internet. we use 8-bit compression that reduces the resource requirements to run very large models. in addition, we develop algorithms for reliable routing and load balancing. with the release of this system, we hope to broaden access to llms and pave the road to applications, studies or research questions that were previously not possible or simply too expensive. running llms over the internet raises a broad range of related questions. one of them is privacy: how to avoid revealing private data to outside peers. another challenge is to ensure that participants can benefit from this system equitably, i.e. in proportion to their contribution. we discuss future problems such as privacy, security, and incentive structures in appendix a.