O pen D elta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models (2023)

Abstract: The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as “delta tuning” in Ding et al. (2022), which updates only a small subset of parameters, known as “delta modules”, while keeping the backbone model’s parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs’ code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.

## introduction with the rapid development of self-supervised learning methods in the realm of deep learning, especially pre-training techniques @xcite @xcite , foundational pre-trained models @xcite (ptms) have become a common cornerstone for numerous downstream tasks. and as a result, research into large-scale ptms has flourished. nevertheless, the ever-expanding scale of ptms also poses substantial obstacles in practical use. in traditional model adaptation, all the parameters of the ptms are optimized for each downstream task, which becomes increasingly impractical as the model scales. firstly, optimizing all the parameters incurs prohibitive computing and memory consumption; secondly, storing a fine-tuned model instance for each task or experiment significantly amplifies the storage cost. to address these challenges, researchers have developed parameter-efficient methods for model adaptation. such methods keep the parameters of the main model fixed and update only a small subset of parameters during adaptation. this approach, known as "delta tuning", is described and surveyed in @xcite . different delta tuning methods have been proposed, with varying types and positions of "delta modules". for example, adapter module @xcite is composed of two low-dimensional linear projection layers with an activation function, while lora @xcite module introduces a low-rank decomposition for the weight matrix. bitfit @xcite , on the other hand, specifies the bias vector in ptms as the delta modules. the delta module can be applied to different positions @xcite @xcite to achieve either better performance or efficiency. theoretically, incorporating most delta tuning methods would necessitate restructuring the backbone model, a requirement conventionally achieved through direct code manipulation. while this method may seem simple, it carries several disadvantages. primarily, it lacks flexibility, as delta modules can theoretically be implemented in various positions, making modifications to each position in the backbone model code a cumbersome task. additionally, this method is not scalable, as accommodating delta tuning for newly introduced ptms requires fresh code modifications, posing a challenge for researchers and engineers. in this paper, we present a novel approach to implement delta tuning methods. our approach modifies the backbone model's architecture after it is loaded into the memory. we propose four essential techniques, namely named-based addressing, dynamic tensor re-routing, runtime initialization, and a visualization system. using these key techniques, we build opendelta, an open-source toolkit for delta tuning without modifying the backbone model code. opendelta has several key features. firstly, it is simple to use. migrating from existing full-parameter training to delta tuning requires as few as three lines of code. for beginners or engineers, we also support automatic delta model construction. secondly, it is modular, with delta modules implemented as independent sub-modules that can be attached to or detached from the backbone models. this feature allows different delta modules to coexist and cooperate in the same backbone model and serves multiple tasks flexibly. thirdly, opendelta is highly extensible, supporting pre-trained models in a wide range of frameworks, including both official implementations from the huggingface library @xcite and customized ptms. it can potentially be used with newly emerged ptms and integrated with other ptms' frameworks for efficient training, such as the parallel training framework. ## related work our work is related to delta tuning, more specifically, the implementation of delta tuning methods. ## motivation in this section, we begin by presenting the unified formulation of delta tuning. then we underscore a set of crucial characteristics of delta tuning, focusing on the implementation aspect, which emphasizes the pressing need for a novel toolkit to aid in the research and advancement of delta tuning approaches. ## opendelta in light of the aforementioned key features of delta tuning, we present opendelta. we will begin by presenting an overview of opendelta. following that, we will delve into the key implementations of this framework. ## conclusion in summary, opendelta is a plug-and-play library for delta tuning, offering an intuitive and modular solution to adapt large ptms using delta tuning without the need for code modifications. the library's user-friendliness, flexibility, and extensibility make it accessible and useful for both researchers and engineers. in the future, we plan to continuously update the library with new delta tuning methods and ensure its compatibility with the latest versions of other major ptms libraries.