A dapter H ub: A Framework for Adapting Transformers (2020)

Abstract: The current modus operandi in NLP involves downloading and fine-tuning pre-trained models consisting of millions or billions of parameters. Storing and sharing such large trained models is expensive, slow, and time-consuming, which impedes progress towards more general and versatile NLP methods that learn from and for many tasks. Adapters—small learnt bottleneck layers inserted within each layer of a pre-trained model— ameliorate this issue by avoiding full fine-tuning of the entire model. However, sharing and integrating adapter layers is not straightforward. We propose AdapterHub, a framework that allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. The framework, built on top of the popular HuggingFace Transformers library, enables extremely easy and quick adaptations of state-of-the-art pre-trained models (e.g., BERT, RoBERTa, XLM-R) across tasks and languages. Downloading, sharing, and training adapters is as seamless as possible using minimal changes to the training scripts and a specialized infrastructure. Our framework enables scalable and easy access to sharing of task-specific models, particularly in low-resource scenarios. AdapterHub includes all recent adapter architectures and can be found at AdapterHub.ml

## introduction recent advances in nlp leverage transformerbased language models @xcite , pretrained on large amounts of text data @xcite @xcite . these models are fine-tuned on a target task and achieve state-of-the-art (sota) performance for most natural language understanding tasks. their performance has been shown to scale with their size @xcite and recent models have reached * *equal contribution. adapters @xcite have been introduced as an alternative lightweight fine-tuning strategy that achieves on-par performance to full fine-tuning @xcite on most tasks. they consist of a small set of additional newly initialized weights at every layer of the transformer. these weights are then trained during fine-tuning, while the pre-trained parameters of the large model are kept frozen/fixed. this enables efficient parameter sharing between tasks by training many task-specific and language-specific adapters for the same model, which can be exchanged and combined post-hoc. adapters have recently achieved strong results in multi-task and cross-lingual transfer learning @xcite . however, reusing and sharing adapters is not straightforward. adapters are rarely released individually; their architectures differ in subtle yet important ways, and they are model, task, and language dependent. to mitigate these issues and facilitate transfer learning with adapters in a range of settings, we propose adapterhub, a framework that enables seamless training and sharing of adapters. adapterhub is built on top of the popular transformers framework by huggingface foot_0 @xcite , which provides access to stateof-the-art pre-trained language models. we en-hance transformers with adapter modules that can be combined with existing sota models with minimal code edits. we additionally provide a website that enables quick and seamless upload, download, and sharing of pre-trained adapters. adapter-hub is available online at: adapterhub.ml. adapterhub for the first time enables nlp researchers and practitioners to easily and efficiently share and obtain access to models that have been trained for particular tasks, domains, and languages. this opens up the possibility of building on and combining information from many more sources than was previously possible, and makes research such as intermediate task training @xcite , composing information from many tasks @xcite , and training models for very low-resource languages @xcite ) much more accessible. ## adapters while the predominant methodology for transfer learning is to fine-tune all weights of the pre-trained model, adapters have recently been introduced as an alternative approach, with applications in computer vision @xcite as well as the nlp domain @xcite @xcite . ## adapterhub adapterhub consists of two core components: 1) a library built on top of huggingface transformers, and 2) a website that dynamically provides analysis and filtering of pre-trained adapters. adapterhub provides tools for the entire life-cycle of adapters, illustrated in figure 1 trained transformer model θ; } performing inference with the trained adapter transformer model. we minimize the required changes to existing huggingface training scripts, resulting in only two additional lines of code. in figure 2 we present the required code to add adapter weights (line 3) and freeze all the transformer weights θ (line 4). in this example, the model is prepared to train a task adapter on the binary version of the stanford sentiment treebank (sst; @xcite using the adapter architecture of @xcite . similarly, language adapters can be added by setting the type parameter to adaptertype.text language, and other adapter architectures can be chosen accordingly. while we provide ready-made configuration files for well-known architectures in the current literature, adapters are dynamically configurable, which makes it possible to define a multitude of architectures. we illustrate the configurable components as dashed lines and objects in figure 3 . the configurable components are placements of new weights, residual connections as well as placements of lay-ernorm layers @xcite . the code changes within the huggingface transformers framework are realized through mixins, which are inherited by the respective transformer classes. this minimizes the amount of code changes of our proposed extensions and encapsulates adapters as designated classes. it further increases readability as adapters are clearly separated from the main transformers code base, which makes it easy to keep both repositories in sync as well as to extend adapterhub. when adapter training has completed, the parameter file together with the corresponding adapter configuration file are zipped and uploaded to a public server. the user then enters the metadata (e.g., url to weights, user info, description of training procedure, data set used, adapter architecture, github handle, twitter handle) into a designated yaml file and issues a pull request to the adapter-hub github repository. when all automatic checks pass, the adapterhub.ml website is automatically regenerated with the newly available adapter, which makes it possible for users to immediately find we also provide a configuration file for the architecture proposed by @xcite , not shown here. and use these new weights described by the metadata. we hope that the ease of sharing pre-trained adapters will further facilitate and speed up new developments in transfer learning in nlp. figure 4 : | after the correct adapter has been identified by the user on the explore page of adapterhub.ml, they can load and stitch the pre-trained adapter weights φ into the transformer θ (line 3). ## conclusion and future work we have introduced adapterhub, a novel easy-touse framework that enables simple and effective transfer learning via training and community sharing of adapters. adapters are small neural modules that can be stitched into large pre-trained transformer models to facilitate, simplify, and speed up transfer learning across a range of languages and tasks. adapterhub is built on top of the commonly used huggingface transformers, and it requires only adding as little as two lines of code to existing training scripts. using adapters in adapter-hub has numerous benefits such as improved reproducibility, much better efficiency compared to full fine-tuning, easy extensibility to new models and new tasks, and easy access to trained models. with adapterhub, we hope to provide a suitable and stable framework for the community to train, search, and use adapters. we plan to continuously improve the framework, extend the composition and modularity possibilities, and support other transformer models, even the ones yet to come.