Investigating Learning Dynamics of BERT Fine-Tuning (2020)

Abstract: The recently introduced pre-trained language model BERT advances the state-of-the-art on many NLP tasks through the fine-tuning approach, but few studies investigate how the fine-tuning process improves the model performance on downstream tasks. In this paper, we inspect the learning dynamics of BERT fine-tuning with two indicators. We use JS divergence to detect the change of the attention mode and use SVCCA distance to examine the change to the feature extraction mode during BERT fine-tuning. We conclude that BERT fine-tuning mainly changes the attention mode of the last layers and modifies the feature extraction mode of the intermediate and last layers. Moreover, we analyze the consistency of BERT fine-tuning between different random seeds and different datasets. In summary, we provide a distinctive understanding of the learning dynamics of BERT fine-tuning, which sheds some light on improving the fine-tuning results.

## introduction bert (bidirectional encoder representations from transformers; @xcite ) is a large pre-trained language model. it obtains state-of-theart results on a wide array of natural language processing (nlp) tasks. unlike other previous pretrained language models @xcite , bert employs the multi-layer bidirectional transformer encoder as the model architecture and proposes two novel pre-training tasks: the masked language modeling and the next sentence prediction. there are two approaches to adapt the pretrained language representations to the downstream tasks. one is the feature-based approach, where the parameters of the original pre-trained model are frozen when applied on the downstream tasks @xcite @xcite . another one is the fine-tuning approach, where the pre-trained model and the taskspecific model are trained together @xcite @xcite . take the classification task as an example, the new parameter added for bert fine-tuning is a task-specific fully-connected layer, then all parameters of bert and the classification layer are trained together to minimize the loss function. @xcite demonstrate that the finetuning approach of bert generally outperforms the feature-based approach. we know that bert encodes task-specific representations during finetuning, but it is unclear about the learning dynamics of bert fine-tuning, i.e., how fine-tuning helps bert to improve performance on downstream tasks. we investigate the learning dynamics of bert fine-tuning with two indicators. first, we use jensen-shannon divergence to measure the change of the attention mode during bert fine-tuning. second, we use singular vector canonical correlation analysis (svcca; @xcite ) distance to measure the change of the feature extraction mode. we conclude that during the fine-tuning procedure, bert mainly changes the attention mode of the last layers, and modifies the feature extraction mode of intermediate and last layers. at the same time, bert has the ability to avoid catastrophic forgetting of knowledge in low layers. moreover, we also analyze the consistency of the fine-tuning procedure. across different random seeds and different datasets, we observe that the changes of low layers (0-9th layer) are generally consistent, which indicates that bert has learned some common transferable language knowledge in low layers during the pre-training process, while the task-specific information is mostly encoded in intermediate and last layers. ## experimental setup we employ the bert-large model 1 on a diverse set of nlp tasks: natural language inference (nli), sentiment analysis (sa) and paraphrase detection (pd). for nli, we use both the multi-genre natural language inference dataset (mnli; @xcite ) and the recognizing textual entailment dataset (rte; aggregated from @xcite @xcite . for sa, we use the binary version of the stanford sentiment treebank dataset (sst-2; @xcite . for pd, we use the microsoft research paraphrase corpus dataset (mrpc; @xcite . the hyperparameter choice for fine-tuning is task-specific. we choose relatively optimal parameters for every dataset as suggested in @xcite . the detailed hyperparameter configuration is shown in ## fine-tuning changes the attention mode of the last layers the model architecture of bert is essentially based on the multi-layer bidirectional transformer, the core function of which is the self-attention mechanism @xcite . we use jensen-shannon divergence between two attention scores to detect changes of the attention mode in different layers during fine-tuning. jensen-shannon divergence js divergence is a method of measuring the distance between two 1 github.com/google-research/bert probability distributions, it is defined as: for every layer of bert, there are 16 attention heads, each head produces an attention score of the input sequence. each attention score is a probability distribution about how much attention a target word pays to other words. we compute js divergence of attention scores between the original bert model m 0 and the fine-tuned model m t on the development set, by calculating the average of the sum of js divergence at each word and each attention head for every layer, the specific calculation formula is as follows: ## fine-tuning modifies the feature extraction mode of the intermediate and the last layers while the attention score implies the inherent dependencies between different words, the output representation of every layer is the practical feature that the model extracts. we use svcca distance @xcite to quantify the change of these output representations during fine-tuning, which indicates the change of the feature extraction mode of bert. singular vector canonical correlation analysis. svcca distance is used as a metric to measure the differences of hidden representations between the original bert model m 0 and the finetuned model m t at a target layer. it is calculated by: as the output representation of the last layer is directly used for classification, we aim to compare the effect of each layer on the final output representation respectively. we replace the parameters of /d\hu,qgh[ 667 053& 01/, every layer in the fine-tuned model with their original values in the bert model before fine-tuning and compute the svcca distance of the last layer output representation. the results are shown in figure 3 , we observe that whether the low layers (0-10) are replaced with their original values or not, it has little effect on the final output representation. moreover, the change in the intermediate and last layers will increase the svcca distance, which reflects that fine-tuning mainly changes the feature extraction mode of intermediate and last layers. ## consistency of fine-tuning in this section, we investigate the consistency of different fine-tuning procedures, including the consistency between different random seeds and the consistency between different datasets. ## related work pre-trained language models @xcite @xcite @xcite @xcite ) stimulate the research interest on the interpretation of these black-box models. @xcite show that the bilm-based models learn representations that vary with network depth, the lower layers specialize in local syntactic relationships and the higher layers model longer range relationships. @xcite propose a methodology and offer the analysis of berts capacity to capture different kinds of linguistic information by encoding it in its self-attention weights. @xcite visualize the loss landscapes and optimization trajectories of the bert fine-tuning procedure and find that low layers of the bert model are more invariant and transferable across tasks. @xcite find that fine-tuning primarily affects the top layers of bert, but with noteworthy variation across tasks. @xcite propose a self-attention attribution method to interpret information flow within transformer. ## discussions we use js divergence to detect the change of the attention mode in different layers during bert fine-tuning and use svcca distance to detect the change of the feature extraction mode. we observe that bert fine-tuning mainly changes the attention mode of last layers and modifies the feature extraction mode of intermediate and last layers. we also demonstrate that the changes of low layers are consistent between different random seeds and different datasets, which indicates that bert learns common transferable language knowledge in low layers. in future research, we would like to explore learning dynamics for cross-lingual pretrained models @xcite @xcite .