Robust Transfer Learning with Pretrained Language Models through Adapters (2021)

Abstract: Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.

## introduction pretrained transformer-based language models like bert @xcite , roberta @xcite have demonstrated impressive performance on various nlp tasks such as sentiment analysis, question answering, text generation, just to name a few. their successes are achieved through sequential transfer learning @xcite : pretrain a language model on large-scale unlabeled data and then fine-tune it on downstream tasks with labeled data. the most commonly used fine-tuning approach is to optimize all parameters of the pretrained model with regard to the downstream-task-specific loss. this training scheme is widely adopted due to its simplicity and flexibility @xcite @xcite @xcite @xcite . despite the success of the standard sequential transfer learning approach, recent works @xcite @xcite have explored domain-specific or taskspecific unsupervised pretraining, that is, masked language model training on the downstream task data before the final supervised fine-tuning on it. and they demonstrated benefits of task-specific pretraining on transfer learning performance. however, both standard sequential transfer learning and that with task-specific pretraining are unstable in the sense that downstream task performance is subject to considerable fluctuation while the random seed is changed or the number of pretraining and/or fine-tuning iterations is varied even after the training has converged (see section 2 and section 3 for details). for instance, as observed in fig. 1 , as the number of task-specific pretraining iteration varies, cola's performance is severely unstable in fine-tuning. besides instability, we also observe that task-specific pretraining is vulnerable to adversarial attack. last but not least, task-specific pretraining and/or fine-tuning on the entire model is highly parameter-inefficient given the large size of these models (e.g., the smallest bert has 110 million parameters). in this work, we propose a simple yet effective adapter-based approach to mitigate these issues. adapters are some small bottleneck layers inserted within each layer of a pretrained model @xcite . the adapter layers are much smaller than the pretrained model in terms of the number of parameters. for instance, the adapter used in @xcite only adds 3.6% parameters per task. in our approach, we adapt the pretrained model to a downstream task through 1) task-specific pretraining and 2) taskspecific supervised training (namely, fine-tuning) on the downstream task (e.g., classification, sequence labeling) by only optimizing the adapters and keeping all other layers fixed. our approach is parameter-efficient given that only a small number of parameters are learned in the adaptation. the adapted model learned through our approach can be viewed as a residual form of the original pretrained model. suppose x is an input sequence and h original is the features of x computed by the original model. then the feature computed by the adapted model is, some prior works have examined the potential causes of the instability of pretrained language models in transfer learning. @xcite proposed that catastrophic forgetting in sequential transfer learning underlined the instability, while @xcite proposed that gradient vanishing in fine-tuning caused it. pinpointing the cause of transfer learning instability is not the focus of the current work, but our proposed method seems to able to enhance transfer learning on both aspects. the standard sequential transfer learning or that with task-specific pretraining updates all model parameters in fine-tuning. in contrast, our approach keeps the pretrained parameters unchanged and only updates the parameters in the adapter layers, which are a small amount compared to the pretrained parameters. therefore, our approach naturally alleviates catastrophic forgetting considering the close distance between the original pretrained model and the adapted model. on the other hand, we do not observe gradient vanishing with our transfer learning scheme (see section 2 for more details). this might be because optimizing over a much smaller parameter space in our approach, compared to the standard sequential transfer learning scheme where all parameters are trained, renders the optimization easier. we leave it to future work for further theoretical analysis. in addition to its improved stability, the proposed transfer learning scheme is also likely to be more robust to adversarial attack. given that it updates the entire model, the standard transfer learning approach might suffer from overfitting to the downstream task, and thus a small perturbation in the input might result in consequential change in the model prediction. in turn, it might be susceptible to adversarial attack. our approach only updates a much smaller portion of parameters, and hence might be more robust to these attacks, which is confirmed in our empirical analysis (see section 4). contributions. in summary our work has the following contributions. ## instability to different random seeds we first evaluate the training instability with respect to multiple random seeds: fine-tuning the model multiple times in the same setting, varying only the random seed. we conduct the experiments on rte @xcite when fine-tuning 1) bert-base-uncased @xcite and 2) bert-base-uncased with the adapter @xcite foot_1 . as shown in figure 2 , the model without adapter leads to a large standard deviation on the fine-tuning accuracy, while the one with adapter results in a much smaller variance on the task performance. mosbach et al. (2020) argues that the fine-tuning instability can be explained by optimization difficulty and gradient vanishing. in order to inspect if the adapter-based approach suffers from this optimization problem, we plot the l 2 gradient norm with respect to different layers of bert, pooler layer and classification layer, for fine-tuning with or without adapter in in traditional fine-tuning (without adapter), we see vanishing gradients for not only the top layers but also the pooler layer and classification layer. this is in large contrast to the with-adapter finetuning. the gradient norm in the with-adapter finetuning does not decrease significantly in the training process. these results imply that the adaptation with adapter does not exhibit gradient vanishing and presents a less difficult optimization problem, which in turn might explain the improved stability of our approach. ## instability to pretraining and fine-tuning iterations fine-tuning with all parameters also exhibits another instability issue. in particular, fine-tuning a model multiple times on the pretrained language model, varying the task-specific pretraining iterations and fine-tuning iterations, leads to a large standard deviation in downstream task performance. as observed in figure 1 , cola's performance when varying the task-specific pretraining iterations is severely unstable during pretraining iterations and fine-tuning iterations. the model has converged at the pretraining iteration of 8000. however, finetuning based on this model does not obtain the best performance. pretraining iterations. figure 4 displays the performance on cola of 10 fine-tuning runs with and without the adapter. for each run, we vary only the number of pretraining iterations from 2000 to 20000 with an interval of 2000 and fix the finetuning epochs to 10. we clearly observe that most runs for bert with adapter outperforms the one without adapter. moreover, the adapter makes pretraining bert significantly more stable than the standard approach (without adapter). fine-tuning iterations. we then study the stability with regard to the number of fine-tuning iterations. we show box plots for bert using various pretraining iterations and fine-tuning iterations, with and without adapter in figure 9 . the three sub-figures represent the early, mid, and late stages of pretraining, corresponding to the 0-th, 10000-th, and 20000-th iteration respectively. the 0-th iteration represents the original model without task-specific pretraining. the model suffers from underfitting in the 0-th iteration and overfitting in the 20000-th iteration. in figure 9 (a), we plot the distributions of the development scores from 100 runs when fine-tuning bert with various fine-tuning epochs ranging from 1 to 100. in the early stage, the average development score of the model with the adapter is a little lower than the baseline model while the stability is better. after several epochs of pretraining, the adapter gradually shows improved performance in terms of the mean, minimum and maximum as demonstrated in figure 9 (b) . in the end of the pretrainig, there exists an over-fitting problem for the traditional bert models. pretraining transfers the model to a specific domain and fails to maintain the original knowledge. in contrast, the performance with the adapter still grows as training continues and consistently benefit from pretraining. besides, we observe that the adapter leads to a small variance in the fine-tuning performance, especially in the late stage. additional plots and learning curves can be found in the appendix. ## adversarial robustness while successfully applied to many domains, the predictions of transformers @xcite become unreliable in the presence of small adversarial perturbations to the input @xcite . therefore, the adversarial attacker has become an important tool @xcite to verify the robustness of models. the robustness is usually evaluated from attack effectiveness (i.e., attack success rate). we use a sota adversarial attack approach to assess the robustness: pwws attacker (ren et al., 2019). 4 . figure 5 shows the attack success rate of bert with/without adapter during task-specific pretraining on sst-2. the x-axis is the number of epochs for task-specific pretraining. it can be observed that the model with the adapter has better adversarial robustness. ## conclusion we propose a simple yet effective transfer learning scheme for large-scale pretrained language model. we insert small bottleneck layers (i.e., adapter) within each block of the pretrained model and then optimize the adapter layers in task-specific unsupervised pretraining and supervised training (i.e., fine-tuning) while fixing the pretrained layers. extensive experiments demonstrate that our approach leads to improved stability with respect to different random seeds and different number of iterations in task-specific pretraining and fine-tuning, enhanced adversarial robustness, and better transfer learning task performance. we therefore consider the proposed training scheme as a robust and parameterefficient transfer learning approach.