Learned Adapters Are Better Than Manually Designed Adapters (2023)

Abstract: Recently, a series of works have looked into further improving the adapter-based tuning by manually designing better adapter architectures. Understandably, these manually designed solutions are sub-optimal. In this work, we propose the Learned Adapter framework to automatically learn the optimal adapter architectures for better task adaptation of pre-trained models (PTMs). First, we construct a unified search space for adapter architecture designs. In terms of the optimization method on the search space, we propose a simple-yet-effective method, GDNAS for better architecture optimization. Extensive experiments show that our Learned Adapter framework can outperform the previous parameter-efficient tuning (PETuning) baselines while tuning comparable or fewer parameters. Moreover: (a) the learned adapter architectures are explainable and transferable across tasks. (b) We demonstrate that our architecture search space design is valid.

## introduction increasingly large pre-trained models @xcite @xcite @xcite built upon the transformer architecture @xcite have been emerging and achieving the state-of-the-art (sota) results on a variety of downstream tasks @xcite @xcite @xcite @xcite @xcite . despite their effectiveness, these large-scale models also bring the curse of prohibitive computation @xcite @xcite and storage costs during the adaptations to downstream tasks due to the gradient computation of the whole model and the giant size of the fine-tuned checkpoint. recently, parameter efficient tuning (petuning) has raised much attention in the research field since it can only train a small portion of ptms and keep the vast majority of parameters frozen, thus alleviating the computation costs during full finetuning. a series of studies @xcite @xcite @xcite @xcite has verified that petuning can achieve competitive performance compared to conventional finetuning with very few trainable parameters, resulting in a considerable reduction in model adaptation costs. adapter-based methods @xcite @xcite inject newly-introduced layers after or around the attention or feed-forward modules of the transformer block, and yield promising results by fine-tuning a small portion of the ptm's parameters. recently, a branch of recent research has advanced the understanding of adapter-based tuning more deeply and improved the adapters' architectures to improve parameter efficiency further. adaptable adapters @xcite propose that adapters at different layers should have different activation functions. thus they fit the rational activation functions to downstream tasks during parameter tuning. adapterdrop @xcite tries to reduce the number of adapters' parameter number by not inserting adapters on the lower layers. @xcite bridge connections among different petuning approaches to form a unified framework and further propose to insert adapters in parallel to the modules of the transformer block. @xcite and @xcite propose to add encoding operations between the projection layers of an adapter and achieve better petuning performances. the above empirical evidence implies that altering the adapters' architecture designs can help to improve the petuning performances of adapters with even fewer tunable parameters. predictably, such an optimal architecture is difficult to construct manually and may vary across different ptm backbones and tasks. therefore, we propose to search for the optimal architecture of adapters automatically. we present the learned adapter framework to search for the optimal architecture of adapters automatically. we first construct a unified search space (figure 2 ) that considers various design choices of adapters, including the activation functions, encoding operations, and how the adapters are connected to the ptm backbone. in terms of the specific methodology for optimization on the search space, make a simple-yet-effective modification to the optimization method in darts @xcite , which is better at identifying the proper components for adapters at different intermediate layers. we conduct extensive experiments to study the effectiveness of our learned adapter framework. the experimental results show that with 0.068% parameters, we can recover 99.5% finetuning performances on the glue @xcite benchmark. moreover, the searched architecture outperforms the manually designed petuning baselines while tuning fewer parameters. figure 1 depicts the overall comparison between our learned adapter and the baselines. furthermore, the learned architectures of adapters are transferable across tasks, which significantly strengthens the usefulness of the searched structures. further experiments demonstrate that our newly proposed search space for adapters is valid. ## related work adapter-based tuning. one of the most important research lines of petuning is adapter-based tuning. adapter @xcite inserts adapter modules with bottleneck architecture between every consecutive transformer @xcite sublayers. adapterfusion @xcite only inserts sequential adapters after the feed-forward module. adapter-based tuning methods have comparable results with model tuning when only tuning a fraction of the backbone model's parameter number. due to their amazing results on petuning, a branch of literature has investigated the architecture of adapters in search of further improvements. @xcite analyze a wide range of petuning methods and show that they are essentially equivalent. they also propose the general architecture of petuning. adapterdrop @xcite investigates the efficiency of removing adapters from lower layers. adaptive adapters @xcite investigate the activation functions of adapters and propose to learn the activation functions of adapters via optimizing the parameters of rational functions as a part of the model parameters. compacter @xcite uses lowrank parameterized hypercomplex multiplication @xcite to compress adapters' tunable parameters. there is also work @xcite trying to add different encoding operations, like self-attention operations and convolutions between the bottleneck structure of adapters, and achieve better performances. our work complements this branch of literature by investigating: (a) whether and how the adapter architectures affect the petuning performances, and whether different layers of ptms need different adapter architectures; (b) whether we can obtain better adapter architectures via neural architecture search. other petuning methods another main research line of petuning is the prompt-based tuning that inserts some additional soft prompts into the hidden states instead of injecting new neural modules to ptms. prompt tuning @xcite and p-tuning @xcite insert a soft prompt to word embeddings only, and can achieve competitive results when applied to supersized ptms. prefix-tuning @xcite and p-tuning v2 @xcite insert prompts to every hidden layer of ptm. idpg @xcite uses the prompt generator with parameterized hypercomplex multiplication @xcite to generate a soft prompt for every instance. there are also some other popular petuning methods, such as bitfit @xcite which only tunes the bias terms, lora @xcite which optimizes low-rank decomposition matrices of the weights within self-attention layers. neural architecture search in the early attempts, neural architecture search (nas) requires massive computations, like thousands of gpu days @xcite @xcite . recently, a particular group of one-shot nas, led by the seminal work darts @xcite has attracted much attention. darts formulates the search space into a super-network that can adjust itself in a continuous space so that the network and architectural parameters can be optimized alternately (bi-level optimization) using gradient descent. a series of literature try to improve the performance and efficiency of darts, such as xie et al. (2019), chen et al. (2021), chu et al. (2021) , @xcite . snas @xcite reformulate darts as a credit assignment task while maintaining the differentiability. @xcite penalize the entropy of the architecture parameters to encourage discretization on the hyper-network. p-darts @xcite analyze the issues during the darts bi-level optimization, and propose a series of modifications. pc-darts @xcite reduces the memory cost during search by sampling a portion of the channels in supernetworks. fairdarts @xcite change the softmax operations in darts into sigmoid and introduce a zero-one loss to prune the architectural parameters. xnas @xcite dynamically wipes out inferior architectures and enhances superior ones. nas is widely applied in both computer vision and natural language processing, especially in knowledge distillation @xcite . our work complements the literature by examining the optimization of darts on our search space and propose a new training procedure that does not require re-training after discretization. ## conclusion in this work, we propose the learned adapter framework, which automatically optimizes the adapter architectures. first, we design a unified search space for adapters, taking into account the recent works of manual adapter designs. second, in light of the issues in the darts method, we propose a novel gdnas method that can deliver better adapter architectures and requires no re-training of the learned adapter architectures. we run extensive experiments and analyses on the glue benchmark, demonstrating that our learned adapter framework can achieve better tuning performances than the baselines while maintaining parameter efficiency.