Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning (2023)

Abstract: We introduce Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models. By integrating 10 diverse adapter methods into a unified interface, Adapters offers ease of use and flexible configuration. Our library allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex adapter setups. We demonstrate the library’s efficacy by evaluating its performance against full fine-tuning on various NLP tasks. Adapters provides a powerful tool for addressing the challenges of conventional fine-tuning paradigms and promoting more efficient and modular transfer learning. The library is available via https://adapterhub.ml/adapters.

## introduction the ever-increasing size of pretrained large language models (llms) @xcite has made the established transfer learning paradigm of fine-tuning all model parameters on a downstream task @xcite extremely expensive. moreover, the requirement of parameter efficiency at fine-tuning, while definitely paramount, is not the only shortcoming of the predominant llm finetuning paradigm. it also suffers from other crucial issues such as negative interference, lack of positive transfer between tasks in multi-task learning @xcite , catastrophic forgetting @xcite , and poor generalization. two closely related lines of research aimed at addressing this set of challenges have gained significant attention recently. first, parameter-efficient fine-tuning @xcite focuses on the aspect of computational efficiency and feasibility by only fine-tuning a small * authors contributed equally. number of parameters for downstream tasks. second, modular transfer learning @xcite focuses on the aspect of knowledge transfer by designing self-contained network modules which can be aggregated for better multi-task performance and generalization. in practice, these often represent two sides of the same coin. methods that devise small components within a language model for fine-tuning on labeled task data, henceforth generally denoted as adapters, are both parameter-efficient and modular in nature. the initial release of adapterhub @xcite marks the first attempt to systematically make adapters accessible to researchers and practitioners in an easy-to-use framework. adapterhub proposed a framework to easily integrate, train and use adapters for state-of-the-art transformer models with minimal changes. it additionally established an open platform to share, explore and consume pre-trained adapter modules. while adapter-hub focused on bottleneck-style adapters @xcite initially, the field of adapter methods has expanded substantially since @xcite @xcite @xcite . with increasing interest in adapter methods, new tools and libraries have been developed. opendelta @xcite ), huggingface's peft @xcite and llm-adapters @xcite are recent examples of libraries which attempt to unify adapter methods in a single code base and extend their applicability to new model architectures. however, these works exclusively focus on the parameter-efficiency aspect of adapters, neglecting the modularity side of these methods. ## background we use the term adapter in a more general sense to refer to a broad family of transfer learning methods that share the two defining properties: parameter efficiency and modularity. for a detailed overview of different adapter architectures, we refer the reader to the recent survey by @xcite . ## the adapters library adapters builds on many design decisions established in the initial adapterhub release @xcite , but offers substantial extensions both 'horizontally' (e.g., extending the support to many more pretrained neural architectures, extending the coverage of adapter architectures) and 'vertically' (e.g., adding new composition and processing capabilities). listing 1: example of adding adapters to an existing transformers model. after model instantiation, init() introduces adapter-specific functionality. two bottleneck adapters are added via add_adapter() and activated for parallel training. ## adapter evaluation in addition to the ease of use aforementioned, we show that the adapter methods offered by our library are performant across a range of settings. to this end, we conduct evaluations on the single adapter implementations made available by adapters (see §3.3.1). we demonstrate the effectiveness of these methods against full fine-tuning on a variety of task types: extractive question answering @xcite , multiple choice classification @xcite , sequence tagging @xcite , sequence to sequence summarization @xcite , sequence classification and sequence regression @xcite . to make our evaluations reflective of user experience, we conduct them using the two most commonly used base model variants on adapterhub: roberta-base @xcite and bart-base @xcite . ## conclusion we have presented adapters, a novel library for research and application of adapters. unlike comparable solutions, adapters equally focuses on the parameter-efficiency and modularity side of adapters. our library implements a diverse set of adapter methods under a unified interface which allows flexible configuration and mixing of different approaches. we proposed a simple building block system for leveraging the modularity of adapters to build complex adapter setups. adapters tightly integrates into the huggingface and adapterhub ecosystems and its adapter implementations show performances competitive to full fine-tuning. as research on adapters and llms continues to advance rapidly, our library will evolve as well. its extensibility makes adapters well prepared for the integration of new adapter methods and model architectures, both from us and the community.