Adaptable Adapters (2022)

Abstract: State-of-the-art pretrained NLP models contain a hundred million to trillion parameters. Adapters provide a parameter-efficient alternative for the full finetuning in which we can only finetune lightweight neural network layers on top of pretrained weights. Adapter layers are initialized randomly. However, existing work uses the same adapter architecture—i.e., the same adapter layer on top of each layer of the pretrained model—for every dataset, regardless of the properties of the dataset or the amount of available training data. In this work, we introduce adaptable adapters that contain (1) learning different activation functions for different layers and different input data, and (2) a learnable switch to select and only use the beneficial adapter layers. We show that adaptable adapters achieve on-par performances with the standard adapter architecture while using a considerably smaller number of adapter layers. In addition, we show that the selected adapter architecture by adaptable adapters transfers well across different data settings and similar tasks. We propose to use adaptable adapters for designing efficient and effective adapter architectures. The resulting adapters (a) contain about 50% of the learning parameters of the standard adapter and are therefore more efficient at training and inference, and require less storage space, and (b) achieve considerably higher performances in low-data settings.

## introduction recent improvements in nlp are heavily skewed towards using larger pretrained models @xcite and given their considerably better performances, using them is becoming unavoidable @xcite . their improvements, however, come at the cost of significant computational resources at training and inference times. for ## evaluation the number of inference layers for adapterdrop aa are equivalent to the number of layers in aa-focused spec experiments for each task and data setting. the number of layers for adapterdrop 13 is 13, which is the same as aa-focused uni and aa-focused sim . note that the number of layers for aa-focused experiments are the same both at training and inference while it is not the case for adapterdrop. the |aa| rows in evaluating the impact of aa on selecting the number of beneficial layers. in the results of to investigate whether the improvements of aafocused sim over the baseline are only due to using a fewer number of adapter layers, we report the results of an adapter architecture in which only the last n adapter layers are added to the model, e.g., for @xmath0. ## conclusion in this paper, we propose adaptable adapters. they consist of a learnable switch to select a subset of beneficial adapter layers and a learnable activation function to learn the suitable activation at each adapter layer and for each input data. the results of adaptable adapters show that we can achieve on-par performances with the full adapter architecture by using a smaller subset of layers. we show that adaptable adapters are viable tools for designing efficient and effective adapter architectures that require less storage space, lower training and inference time with high performances.