AdapterDrop : O n the Efficiency of Adapters in Transformers (2021)

Abstract: Transformer models are expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that AdapterDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.

## introduction while transfer learning has become the go-to method for solving nlp tasks (pan and yang, 2010; torrey and shavlik, 2010; ruder, 2019; howard and ruder, 2018; peters et al., 2018), transformerbased models are notoriously deep requiring millions or even billions of parameters (radford et al., 2018; devlin et al., 2019; radford et al., 2019; liu et al., 2019; brown et al., 2020). this results in slow inference and large storage requirements. at least three independent lines of research have recently evolved to tackle these shortcomings. (1) smaller and faster models that are either distilled or trained from scratch (sanh et al., 2019; sun et al., 2020; bai et al., 2021; wang et al., 2020). (2) robustly trained transformers in which the model depth can be reduced at run-time, thereby decreasing inference time dynamically (fan et al., 2020; elbayad et al., 2020; xin et al., 2020; hou et al., 2020). (3) adapters, which, instead of fully fine-tuning the model, only train a newly introduced set of weights at every layer, thereby sharing the majority of parameters between tasks (houlsby et al., 2019; bapna and firat, 2019; pfeiffer et al., 2020a). adapters have been shown to work well for machine translation (bapna and firat, 2019), cross-lingual transfer (pfeiffer et al., 2020b, 2021b; üstün et al., 2020; vidoni et al., 2020; ansell et al., 2021), community qa (rücklé et al., 2020), and task composition for transfer learning (stickland and murray, 2019; pfeiffer et al., 2021a; lauscher et al., 2020; wang et al., 2021; poth et al., 2021). despite their recent popularity, the computational efficiency of adapters has not been explored beyond parameter efficiency. we close this gap and establish the computational efficiency of two adapter architectures at training and inference time. we investigate different strategies to further improve the efficiency of adapter-based models by incorporating ideas from all three directions mentioned above. our strategies rely on dropping out adapters from transformers, at training and inference time, resulting in models that are dynamically adjustable regarding the available computational resources. our approaches are agnostic to the pre-trained transformer model (e.g., base, large), which makes them broadly applicable. ## efficiency of adapters we first establish the computational efficiency of adapters without adapterdrop. as illustrated in figure 1 , significant differences exist in the forward and backward pass when fine-tuning adapters compared to fully fine-tuning the model. in the forward pass, adapters add complexity with the additional components; however, it is not necessary to backpropagate through the entire model during the backward pass. we compare the training and inference speed of full model fine-tuning against the adapter architectures of houlsby et al. (2019) and pfeiffer et al. (2021a) (depicted in figure 1 ) using the adapterhub.ml framework (pfeiffer et al., 2020a). we conduct our measurements with the transformer configuration of bert base and verify them with different gpus. @xcite we provide measurements corresponding to common experiment configurations in training. adapters can be considerably faster compared to full model fine-tuning-60% faster in some configurations. the two adapter architectures differ only marginally in terms of training efficiency: due to its simpler architecture, training steps of the pfeiffer adapters are slightly faster. the magnitude of the differences depends on the input size; the available cuda cores are the primary bottleneck. @xcite we do not observe any particular differences between adapters and full fine-tuning regarding the training convergence. @xcite the training speedup can be explained by the decreased overhead of gradient computation. most of the parameters are frozen when using adapters and it is not necessary to backpropagate through the first components (see figure 1 ). inference. the two adapter architectures are 94-96% as fast as fully fine-tuned models, which varies depending on the input size. this can have a considerable impact when deployed at scale. ## adapterdrop we have established that adapters are more efficient in terms of training time, however, there is a perpetuate need for sustainable and efficient models (strubell et al., 2019). backpropagating through as few layers as possible would further improve the efficiency of training adapters. the efficiency for inference can be improved by sharing representations at lower transformer layers when simultaneously performing inference for multiple tasks-in other words, when performing multiple independent classifications on the same input. we establish this in motivated by these observations, we propose adapterdrop: dynamically removing adapters from lower transformer layers (depicted in figure 1 ). adapterdrop is similar to dropping out entire transformer layers (fan et al., 2020), however, specialized to adapter settings-where lower layers often have a small impact on the task performances (houlsby et al., 2019). we study two training methods for adapterdrop: (1) specialized adapterdrop: removing adapters from the first n transformer layers, where n is fixed during training. this yields separate models for each possible n. (2) robust adapterdrop: drawing the integer n randomly from [0, 11] for each feed forward add & norm multi-head attention add & norm ff down ff up add & norm emb head ... emb head ... ff down ff up query add & norm softmax adapter adapterfusion add & norm key value emb head ... emb head ... simultaneous tasks 2 4 8 16 speedup (each layer) 4.3% 6.6% 7.8% 8.4% the efficiency gains can be large. when performing inference for multiple tasks simultaneously, we measure inference speedups of 21-42% with five 0 2 4 6 8 dropped layers-depending on the number of simultaneous tasks ( ## efficiency of adapterfusion adapterfusion (pfeiffer et al., 2021a) leverages the knowledge of several adapters from different tasks and learns an optimal combination of the adapters' output representations for a single target task (see figure 3 ). adapterfusion (af) is particularly useful for small training sets where learning adequate models is difficult. despite its effectiveness, af is computationally expensive because all included adapters are passed through sequentially. @xcite ## adapterdrop for adapterfusion there exists considerable potential for improving the efficiency of af, especially at inference time. we address this with two variants of adapterdrop for af by (1) removing entire af layers; (2) pruning the least important adapters from af models. ## conclusion adapters have emerged as a suitable alternative to full model fine-tuning, and their most widely claimed computational advantage is the small model size. in this work, we have demonstrated that the advantages of adapters go far beyond mere parameter efficiency. even without our extensions, the training steps of two common adapter architectures are up to 60% faster. however, these improvements come at the cost of 4-6% slower inference speed. thus, if training is more important, adapters can be advantageous over full model fine-tuning. adapterdrop expands these advantages by dropping a variable number of adapters from lower transformer layers. we dynamically reduce the computational overhead at run-time when performing inference over multiple tasks and maintain task performances to a large extent. this benefits researchers working on models that need to make multiple independent predictions on a single input. finally, we also investigated the computational efficiency of adapterfusion models. we find that dropping entire adapterfusion layers comes at a considerable performance/efficiency tradeoff, whereas pruning of the least activated adapters in each layer can improve the model efficiency while maintaining performance entirely. we believe that our work can be widely extended and that there exist many more directions to obtain efficient adapter-based models. for instance, we could explore more efficient pre-trained adapters, foot_10 sharing the adapter weights across layers, @xcite or pruning adapters from adapterfusion at training time. @xcite in the appendix to this paper, we present preliminary results for several related ideas, which may serve as a starting point for future work.