Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (2022)

Abstract: Fine-tuning over large pretrained language models (PLMs) has established many state-of-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively fine-tuning layers in a top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to the different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple component-wise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. The codebase can be found at https://github.com/yangalan123/FineTuningStability.

## introduction fine-tuning over large pretrained language models (plms), which achieved remarkable performance over various benchmarks, has become the de facto paradigm for several current natural language processing (nlp) systems. however, fine-tuning can be unstable in terms of significant variance in metrics, resulting in even worse-than-random failed models @xcite @xcite . catastrophic forgetting @xcite ) during fine-tuning of plms is one common explanation for this instability @xcite , i.e., plms may lose their rich domain-agnostic knowledge acquired by language model pretraining in the process of fine-tuning. through layer-replacement experiments between pretrained models and finetuned models, @xcite further configure 1 : fine-tuning performance over time on rte datasets. here we fine-tune over bert-large-uncased model @xcite . in each iteration, we train the model for 3 epochs and the errorbar is plotted based on 5 different runs. these findings give rise to a straightforward way to enhance the fine-tuning stability: how about finetuning the model from top to bottom to reduce the parameter changes and hence mitigate the catastrophic forgetting problem? this is reminiscent of the gradual unfreezing @xcite , which does layer-wise top-down fine-tuning and unfreezing new layers only when the layers above have been fine-tuned. therefore, the newly unfrozen layer would only be tuned for a slightly easier optimization problem at each iteration, leading to much fewer changes to the parameters. however, based on a comprehensive case study of the gradual unfreezing method, we obtained empirical observations beyond our expectations ( §2). our analysis further reveals a possible reason: different components (e.g., feed-forward networks at different layers, fully connected matrices and biases at output layer) converge at varying speeds. thus, components in upper layers, which have converged to local optima, cannot easily be fine-tuned with newly unfrozen parameters. based on this observation, we propose a simple component-wise gradient clipping method to stabilize the fine-tuning process ( §3). this method achieves significant empirical improvements of fine-tuning stability in terms of the variance and the failed run percentage over three tasks ( §4). in summary we make the following contributions: ## a bitter case study: layer-wise gradual unfreezing mosbach et al. (2020) attributed the instability problem in fine-tuning process to the catastrophic forgetting in the top layers, through a layer replacement experiment between pretrained and fine-tuned models. following this empirical observation, the instability problem might be mitigated if we can minimize the edits to the pretrained model parameters, especially in top layers. this inspires us to mitigate the instability problem via gradual unfreezing (gu, @xcite ). specifically, suppose we are working with a model m with l layers parameterized by {θ (i) , 1 ≤ @xmath0. gu tunes m for l iterations. at k-th (k start from 0) iteration, we only tune a subset of parameters r in each iteration, we tune the parameter for e epochs, where e is large enough for convergence. 1 detailed algorithm is shown in algorithm 1 at appendix a. failure of gradual unfreezing from fig. 1 , the accuracy of gradual unfreezing is significantly worse than full fine-tuning, foot_1 although it indeed achieves smaller update to pretrained model parameters compared with full fine-tuning (fig. 2a ). convergence racing between parameters to investigate the reason behind this unsatisfying performance of gu, we plot the component-wise maximum update (measured by component-wise maximum rooted mean squared difference, foot_2 as different layers can have multiple components with different dimensions) at each layer in fig. 2b . clearly, from the very beginning, the parameter updates for both early-tuned parameters and newly-unfrozen parameters have quickly diminished in gu so not many updates happen later when new parameters join. based on this observation, we hypothesize that the failure of gu is because the early-tuned parameters have already converged in early iterations and cannot be re-activated later to adapt for newly-unfrozen parameters. to verify this hypothesis, we simply modify gu to stop using early-tuned parameters in the previous iteration and instead copy the weight from the pretrained bert model. for simplicity, we term this method as "gu (restart)". note that full fine-tuning is just the last iteration of gu (restart) when all layers are unfrozen. we plot gu (restart) performance in fig. 1 , and observe that gu (restart) achieves consistently much better performance than gu. however, gu (restart) is much more unstable than gu, and the best performance is only reached when almost all layers have been tuned. ## experiments to evaluate our cwgnc method, we follow @xcite to run our cwgnc method over 25 different runs and report aggregated results on the validation set of three different datasets: rte (acc), mrpc (f1) and cola (mcc). we report the standard deviation (std), averaged performance (mean) and maximum performance (max). hyperparameters are tuned on held-out data. more implementation details are in appendix b. baseline setting for baselines, we first consider the original bert paper reported results @xcite and we run original bert fine-tuning with bias correction to make a stronger baseline following the observation in @xcite . we also compare to mixout regularization methods @xcite and "simple but hard-to-beat baseline" proposed by @xcite . experiment results experiment results are shown in our method is also robust to a wide range of the selection of gradient norm clipping thresholds. we show this in ## conclusion in this paper, we investigate the instability problem of fine-tuning large pretrained language models. inspired by previous works, we first experiment with gradual unfreezing methods, which should help minimize the updates in top layers and ease the catastrophic forgetting problem. however, further experiment results do not support that and we find it is because there is a convergence racing issue between different parameters, namely that earlyconverged parameters can limit the search space for other parameters. based on this finding, we propose to do component-wise gradient norm clipping and achieve significant improvement on averaged performance, smaller standard deviation and quicker convergence speed. our method is robust to the selection of gradient norm clipping threshold. in the future, we will try to study whether the component racing problem also exists in pretrained language models with different sizes and different pretraining methods.