Pruning Adatperfusion with Lottery Ticket Hypothesis (2022)

Abstract: Pre-trained language models have shown great success in multiple downstream tasks. However, they are computationally expensive to fine-tune. Thus, transfer learning with adapter modules has been introduced to alleviate this problem, helping to extract knowledge of the downstream tasks. Adapterfusion models are an example of the transformers-with-adapter-modules, which merge multiple adapters to incorporate knowledge from different tasks. However, merging multiple adapters will inevitably cause redundancies, increasing the training and inference time massively. Therefore, in this paper, we propose an approach to identify the influence of each adapter module and a novel way to prune adapters based on the prestigious Lottery Ticket Hypothesis. Experiments on GLUE datasets show that the pruned Adapterfusion model with our scheme can achieve state-of-the-art results, reducing sizes significantly while keeping performance intact.

## introduction transfer learning with transformer-based pretrained language model has become a go-to method for solving multiple nlp tasks @xcite . the language models are pre-trained on large amounts of unlabeled text data with methods such as masked language modeling (e.g. bert @xcite , roberta @xcite , and xlnet @xcite ). despite the stateof-the-art performance for most natural language understanding tasks, they are notoriously deep requiring millions or even billions of parameters to gain great results @xcite . for different tasks, models needs to be fine-tuned entirely, which is computationally expensive and requires large storage. therefore, adapter modules @xcite are introduced to tackle this issue. it's an alternative way of transfer learning that achieves comparable performance to full fine-tuning on most nlp tasks, without the need of fine-tuning the whole model for a downstream task. adapter is a small residual neural network inserted in each layer of the transformer. during training, only the parameters in adapters are fine-tuned, while the rest of the parameters are frozen. this approach can reduce the number of parameters needed to be trained at the training phase and enable parameter sharing among tasks. moreover, recent studies have revealed that the adapter is capable of extracting knowledge from the target task @xcite , so research attempts have also been made to fuse multiple adapters across multiple tasks to incorporate different aspects of knowledge (e.g. adapterfusion @xcite , k-adapter @xcite ). how-ever, the fusing of adapters in these models can inevitably cause a lot of redundancies. so, @xcite have recently proposed adapterdrop which aims to drop the redundant adapters. they tried to remove adapters from lower transformer layers during training and inferences, resulting in faster training and inference speed with some performance cost. however, the utilization of each adapter has not been fully analysed yet and how to introduce new pruning strategies remains to be explored. to address these deficits, in this paper, we propose an approach to model the utilization of different adapters in the transformer layer, and a novel way to prune adapters in the model while keeping the loss of the performance to be negligible. the contributions are summarized as follows: ## adapterfusion pruning adapterfusion model @xcite is to merge multiple adapters from different tasks. and inference time of the model increases drastically after the fusion. however, not all the adapter modules in the model are utilized in the downstream task. we prune adapter modules from model to improve inference speed. our proposed pruning strategy to remove redundant adapters from adapterfusion model will be two-stage. first, we prune single task adapters before the fusion using lottery ticket hypothesis @xcite . we then fuse the less utilized adapters after training the model. the whole framework is presented in figure 1 . to identify the roles of adapters, we firstly define a new indicator lia (layer influence of adapter) for the adapter module to measure its utilization in each layer. ## experimental studies in this section, we examine the influence of the single task adapter under different downstream tasks, and present the results of our proposed pruning scheme evaluated on the prestigious glue datasets @xcite . ## related work pre-trained language model language models pre-trained on large corpora are widely used in multiple nlp tasks to improve performance. however, these models are often very large. recently, transformer-based @xcite models have become the most popular pre-trained language models. there are plenty of model variants, such as bert @xcite , @xcite , xlnet @xcite , and roberta @xcite , etc. transformer models are huge models, ranging from 110m parameters in bert-base to trillions @xcite in the largest, best-performing models. due to the resource constraints in gpu/tpu memory and computational power, it is difficult to run a large model. so @xcite propose an approach to reduce the amount of training parameters by sharing weights among all transformer layers. the model named albert can lower the usage of memory and speed up the training process of bert. by contrast, albert reduces the amount of parameters needed to be trained, while adapters introduce new parameters and deepen the model. task-rte task-mrpc task-stsb task-cola task-sst2 adapter model is a lightweight fine-tuning approach introduced by @xcite . they insert a small set of newly initialized neural networks named adapters in each layer of the transformers. at training step, only parameters of adapters will be updated and the parameters in the pre-trained language model will be unchanged. therefore it reduces the number of parameters to be trained in the training phase and enables efficient parameter sharing between tasks by combining many task-specific or language-specific adapters. merging and pruning adapters adapters have achieved great results in multi-task @xcite , cross-lingual transfer learning @xcite and infusing knowledge @xcite . adapters are capable of extracting knowledge from different tasks that can be applied to fuse knowledge they learned from different tasks @xcite @xcite . there are plenty of ways to merge multiple adapters, including stacking @xcite , fusing and concatenating adapters @xcite . however, adapters are still far from being concise, and merging multiple adapters from different tasks will introduce redundancies and slow down the inference speed of the model. therefore, @xcite have firstly intro-duced a way to remove adapters from lower transformer layers. by removing the first few layers of the adapters, it effectively speeds up the training and inference of the adapter models. neural networks are easily overparameterized and carry plenty of redundancies. to tackle this problem, distillation @xcite and pruning @xcite are introduced to streamline the model while perserving good performance. and there are several research directions in this field. for pruning before training, mobilenets @xcite is designed for image-recognition networks. for pruning after training, lecun et al. (1990) use the second derivatives to truncate the neural networks. for pruning during training, @xcite reinitialize weights near zeros with random number after training the model. moreover, we can prune models based on activations @xcite , filters @xcite or channels @xcite . the most influential theory recently for pruning comes from @xcite , in which they prove that a dense neural network contains subnetworks (winning ticket) that can have the same performance as the original network when trained isolated. their experiments reveal that not only the structure of the pruned networks matters but also the initial weights of these networks can affect the performance of the model. they also find that a subnetwork extracted from pruning learns faster than the original model and even reaches higher test accuracy. our pruning approach is inspired by this theory and can prune the original adapterfusion model to a much more concise one. ## conclusion and future work in this paper, we propose a new approach to model the utilization of the adapters at each layer by defining a new indicator lia (layer influence of adapter) with which we can identify the most influential adapters. moreover, we introduce a novel way of pruning adapter modules inspired by the prestigious lottery ticket hypothesis. the proposed pruning strategy has been extensively evaluated on the glue datasets, whose results show that we can prune adapters up to 40% of its original size while keeping the performance intact. we further examine the performance and lias of the pruned adapters in the latest state-of-the-art adapterfusion model, and we can remove more than half of the adapters and reduce computation of the adapterfusion model by nearly 40% with little performance loss. this work can be further extended in many ways. for instance, iterative pruning is time-consuming. running 10 iterations of pruning also means training the model for almost 10 times. we can try to find the redundant adapters before and after the training by introducing new elaborate measurements.