SMART : Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020)

Abstract: Transfer learning has fundamentally changed the landscape of natural language processing (NLP). Many state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely high complexity of pre-trained models, aggressive fine-tuning often causes the fine-tuned model to overfit the training data of downstream tasks and fail to generalize to unseen data. To address such an issue in a principled manner, we propose a new learning framework for robust and efficient fine-tuning for pre-trained models to attain better generalization performance. The proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the complexity of the model; 2. Bregman proximal point optimization, which is an instance of trust-region methods and can prevent aggressive updating. Our experiments show that the proposed framework achieves new state-of-the-art performance on a number of NLP tasks including GLUE, SNLI, SciTail and ANLI. Moreover, it also outperforms the state-of-the-art T5 model, which is the largest pre-trained model containing 11 billion parameters, on GLUE.

## introduction the success of natural language processing (nlp) techniques relies on huge amounts of labeled data in many applications. however, large amounts of labeled data are usually prohibitive or expensive to obtain. to address this issue, researchers have resorted to transfer learning. transfer learning considers the scenario, where we have limited labeled data from the target domain for a certain task, but we have relevant tasks * work was done during an internship at microsoft dynamics 365 ai. for many applications in nlp, most popular transfer learning methods choose to pre-train a large language model, e.g., @xcite , gpt @xcite and bert @xcite . such a language model can capture general semantic and syntactic information that can be further used in downstream nlp tasks. the language model is particularly attractive, because it can be trained in a completely unsupervised manner with huge amount of unlabeled data, which are extremely cheap to fetch from internet nowadays. the resulting extremely large multidomain text corpus allows us to train huge language models. to the best of our knowledge, by far the largest language model, t5, has an enormous size of about 11 billion parameters @xcite . for the second fine-tuning stage, researchers adapt the pre-trained language model to the target task/domain. they usually replace the top layer of the language model by a task/domainspecific sub-network, and then continue to train the new model with the limited data of the target task/domain. such a fine-tuning approach accounts for the low-resource issue in the target task/domain, and has achieved state-of-the-art performance in many popular nlp benchmarks @xcite @xcite @xcite . due to the limited data from the target task/domain and the extremely high complexity of the pre-trained model, aggressive fine-tuning often makes the adapted model overfit the training data of the target task/domain and therefore does not generalize well to unseen data. to mitigate this issue, the fine-tuning methods often rely on hyper-parameter tuning heuristics. for example, @xcite use a heuristic learning rate schedule and gradually unfreeze the layers of the language model to improve the fine-tune performance; @xcite give a different suggestion that they only adapt certain layers and freeze the others; @xcite propose to add additional layers to the pre-trained model and fine-tune both of them or only the additional layers. however, these methods require significant tuning efforts. to fully harness the power of fine-tuning in a more principled manner, we propose a new learning framework for robust and efficient fine-tuning on the pre-trained language models through regularized optimization techniques. specifically, our framework consists of two important ingredients for preventing overfitting: (i) to effectively control the extremely high complexity of the model, we propose a smoothnessinducing adversarial regularization technique. our proposed regularization is motivated by local shift sensitivity in existing literature on robust statistics. such regularization encourages the output of the model not to change much, when injecting a small perturbation to the input. therefore, it enforces the smoothness of the model, and effectively controls its capacity @xcite . (ii) to prevent aggressive updating, we propose a class of bregman proximal point optimization methods. our proposed optimization methods introduce a trust-region-type regularization @xcite at each iteration, and then update the model only within a small neighborhood of the previous iterate. therefore, they can effectively prevent aggressive updating and stabilize the finetuning process. we compare our proposed method with several state-of-the-art competitors proposed in @xcite @xcite and show that our proposed method significantly improves the training sta-bility and generalization, and achieves comparable or better performance on multiple nlp tasks. we highlight that our single model with 356m parameters (without any ensemble) can achieve three state-of-the-art results on glue, even compared with all existing ensemble models and the t5 model @xcite , which contains 11 billion parameters. furthermore, we also demonstrate that the proposed framework complements with sota fine-tuning methods @xcite and outperforms the t5 model. we summarize our contribution as follows: 1. we introduce the smoothness-inducing adversarial regularization and proximal point optimization into large scale language model fine-tuning; 2. we achieve state-of-the-art results on several popular nlp benchmarks (e.g., glue, snli, scitail, and anli). notation: we use f (x; θ) to denote a mapping f associated with the parameter θ from input sentences x to an output space, where the output is a multi-dimensional probability simplex for classification tasks and a scalar for regression tasks. π a denotes the projection operator to the set a. d kl (p ||q) = k p k log(p k /q k ) denotes the kl-divergence of two discrete distributions p and q with the associated parameters of p k and q k , respectively. ## background the transformer models were originally proposed in @xcite for neural machine translation. their superior performance motivated @xcite to propose a bidirectional transformer-based language model named bert. specifically, @xcite pre-trained the bert model using a large corpus without any human annotation through unsupervised learning tasks. bert motivated many follow-up works to further improve the pre-training by introducing new unsupervised learning tasks @xcite @xcite , enlarging model size @xcite , enlarging training corpora @xcite @xcite and multi-tasking @xcite . the pre-trained language model is then adapted to downstream tasks and further fine-tuned. specifically, the top layer of the language model can be replaced by a task-specific layer and then continue to train on downstream tasks. to prevent overfitting, existing heuristics include choosing a small learning rate or a triangular learning rate schedule, and a small number of iterations, and other fine-tuning tricks mentioned in @xcite @xcite . our proposed regularization technique is related to several existing works @xcite @xcite . these works consider similar regularization techniques, but target at other applications with different motivations, e.g., semi-supervised learning, unsupervised domain adaptation and harnessing adversarial examples in image classification. our proposed optimization technique covers a large class of bregman proximal point methods in existing literature on optimization, including vanilla proximal point method proposed in @xcite , generalized proximal point method @xcite , accelerated proximal point method, and other variants @xcite @xcite . there is a related fine-tuning method -freelb @xcite , which adapted a robust adversarial training method. however, our framework focuses on the local smoothness, leading to a significant performance improvement. more discussion and comparison are provided in section 4. ## the proposed method we describe the proposed learning framework -smart for robust and efficient fine-tuning of pre-trained language models. our framework consists of two important ingredients: smoothness-inducing adversarial regularization and bregman proximal point optimization foot_0 . ## experiment -main results we demonstrate the effectiveness of smart for fine-tuning large language models using glue @xcite by comparing with existing state-of-the-art methods. dataset details can be found in appendix a. ## experiment -analysis and extension in this section, we first analyze the effectiveness of each component of the proposed method. we also study that whether the proposed method is complimentary to multi-task learning. we further extend smart to domain adaptation and use both snli @xcite and scitail @xcite to evaluate the effectiveness. finally, we verified the robustness of the proposed method on anli @xcite . ## conclusion we propose a robust and efficient computation framework, smart, for fine-tuning large scale pre-trained natural language models in a principled manner. the framework effectively alleviates the overfitting and aggressive updating issues in the fine-tuning stage. smart includes two important ingredients: 1) smooth-inducing adversarial regularization; 2) bregman proximal point optimization. our empirical results suggest that smart improves the performance on many nlp benchmarks (e.g., glue, snli, scitail, anli) with the state-of-the-art pre-trained models (e.g., bert, mt-dnn, roberta). we also demonstrate that the proposed framework is applicable to domain adaptation and results in a significant performance improvement. our proposed fine-tuning framework can be generalized to solve other transfer learning problems. we will explore this direction as future work.