A Closer Look at How Fine-tuning Changes BERT (2022)

Abstract: Given the prevalence of pre-trained contextualized representations in today’s NLP, there have been many efforts to understand what information they contain, and why they seem to be universally successful. The most common approach to use these representations involves fine-tuning them for an end task. Yet, how fine-tuning changes the underlying embedding space is less studied. In this work, we study the English BERT family and use two probing techniques to analyze how fine-tuning changes the space. We hypothesize that fine-tuning affects classification performance by increasing the distances between examples associated with different labels. We confirm this hypothesis with carefully designed experiments on five different NLP tasks. Via these experiments, we also discover an exception to the prevailing wisdom that “fine-tuning always improves performance”. Finally, by comparing the representations before and after fine-tuning, we discover that fine-tuning does not introduce arbitrary changes to representations; instead, it adjusts the representations to downstream tasks while largely preserving the original spatial structure of the data points.

## introduction pre-trained transformer-based language models (e.g., @xcite form the basis of state-of-the-art results across nlp. the relative opacity of these models has prompted the development of many probes to investigate linguistic regularities captured in them (e.g., @xcite @xcite . broadly speaking, there are two ways to use a pre-trained representation @xcite : as a fixed feature extractor (where the pre-trained weights are frozen), or by fine-tuning it for a task. the probing literature has largely focused on the former (e.g., @xcite @xcite @xcite @xcite . some previous work @xcite @xcite does provide insights about fine-tuning: fine-tuning changes higher layers more than lower ones and linguistic information is not lost during fine-tuning. however, relatively less is understood about how the representation changes during the process of fine-tuning and why fine-tuning invariably seems to improve task performance. in this work, we investigate the process of finetuning of representations using the english bert family @xcite . specifically, we ask: first, we find that fine-tuning introduces a divergence between training and test sets, which is not severe enough to hurt generalization in most cases. however, we do find one exception where fine-tuning hurts the performance; this setting also has the largest divergence between training and test set after fine-tuning ( §4.1). second, we examine how fine-tuning changes labeled regions of the representation space. for a representation where task labels are not linearly separable, we find that fine-tuning adjusts it by grouping points with the same label into a small number of clusters (ideally one), thus simplifying the underlying representation. doing so makes it easier to linearly separate labels with fine-tuned representations than untuned ones ( §4.2). for a representation whose task labels are already linearly separable, we find that fine-tuning pushes the clusters of points representing different labels away from each other, thus introducing large separating regions between labels. rather than simply scaling the points, clusters move in different directions and with different extents (measured by euclidean distance). overall, these clusters become distant compared to the untuned representation. we conjecture that the enlarged region between groups admits a bigger set of classifiers that can separate them, leading to better generalization ( §4.3). we verify our distance hypothesis by investigating the effect of fine-tuning across tasks. we observe that fine-tuning for related tasks can also provide useful signal for the target task by altering the distances between clusters representing different labels ( §4.4). finally, fine-tuning does not change the higher layers arbitrarily. this confirms previous findings. additionally, we find that fine-tuning largely preserves the relative positions of the label clusters, while reconfiguring the space to adjust for downstream tasks ( §4.5). informally, we can say that fine-tuning only "slightly" changes higher layers. these findings help us understand fine-tuning better, and justify why fine-tuned representations can lead to improvements across many nlp tasks foot_0 . ## preliminaries: probing methods in this work, we probe representations in the bert family during and after fine-tuning. first, let us look at the two supervised probes we will employ: a classifier-based probe (e.g., @xcite to assess how well a representation supports classifiers for a task, and direct-probe @xcite to analyze the geometry of the representation. ## experimental setup in this section, we describe the representations and tasks we will encounter in our experiments. ## observations and analysis in this section, we will use classifier probes to examine if fine-tuning always improves classifier per-formance ( §4.1). then we propose a geometric explanation for why fine-tuning improves classification performance using directprobe ( §4.2 and §4.3). next, we will confirm this geometric explanation by investigating cross-task finetuning ( §4.4). finally, we will analyze how finetuning changes the geometry of different layers of bert base ( §4.5). ## discussion does fine-tuning always improve performance? indeed, fine-tuning almost always improves task performance. however, rare cases exist where finetuning decreases the performance. fine-tuning introduces a divergence between the training set and unseen examples ( §4.1). however, it is unclear how this divergence affects the generalization ability of representations, e.g. does this divergence suggest a new kind of overfitting that is driven by representations rather than classifiers? how does fine-tuning alter the representation to adjust for downstream tasks? fine-tuning alters the representation by grouping points with the same label into small number of clusters ( §4.2) and pushing each label cluster away from the others ( §4.3). we hypothesize that the distances between label clusters correlate with the classification performance and confirm this hypothesis by investigating cross-task fine-tuning ( §4.4). our findings are surprising because fine-tuning for a classification task does not need to alter the geometry of a representation if the data is already linearly separable in the original representation. what we observe reveals geometric properties that characterize good representations. we do not show theoretical analysis to connect our geometric findings to representation learnability, but the findings in this work may serve as a starting point for a learning theory for representations. how does fine-tuning change the underlying geometric structure of different layers? it is established that higher layers change more than the lower ones. in this work, we analyze this behavior more closely. we discover that higher layers do not change arbitrarily; instead, they remain similar to the untuned version. informally, we can say that fine-tuning only "slightly" changes even the higher layers ( §4.5). nevertheless, our analysis does not reveal why higher layers change more than the lower layers. a deeper analysis of model parameters during fine-tuning is needed to understand the difference between lower and higher layers. limitations of this work. our experiments use the bert family of models for english tasks. given the architectural similarity of transformer language models, we may be able to extrapolate the results to other models, but further work is needed to confirm our findings to other languages or model architectures. in our analysis, we ignore the structure within each cluster, which is another information source for studying the representation. we plan to investigate these aspects in future work. we make our code available for replication and extension by the community. ## related work there are many lines of work that focus on analyzing and understanding representations. the most commonly used technique is the classifierbased method. early work @xcite starts with using linear classifiers as the probe. @xcite pointed out that a linear probe is not sufficient to evaluate a representation. some recent work also employ non-linear probes @xcite . there are also efforts to inspect the representations from a geometric persepctive (e.g. @xcite , including the recently proposed direct-probe @xcite , which we use in this work. another line of probing work designs control tasks @xcite to reverse-engineer the internal mechanisms of representations @xcite . however, in contrast to our work, most studies @xcite @xcite focused on pre-trained representations, not fine-tuned ones. while fine-tuning pre-trained representations usually provides strong empirical performance @xcite , how fine-tuning manage to do so has remained an open question. moreover, the instability @xcite @xcite and forgetting problems @xcite make it harder to analyze fine-tuned representations. despite these difficulties, previous work @xcite @xcite draw valuable conclusions about fine-tuning. this work extends this line of effort and provides a deeper understanding of how fine-tuning changes representations. ## conclusions in this work, we take a close look at how finetuning a contextualized representation for a task modifies it. we investigate the fine-tuned representations of several bert models using two probing techniques: classifier-based probing and direct-probe. first, we show that fine-tuning introduces divergence between training and test set, and in at least one case, hurts generalization. next, we show fine-tuning alters the geometry of a representation by pushing points belonging to the same label closer to each other, thus simpler and better classifiers. we confirm this hypothesis by crosstask fine-tuning experiments. finally, we discover that while adjusting representations to downstream tasks, fine-tuning largely preserves the original spatial structure of points across all layers. taken collectively, the empirical study presented in this work can not only justify the impressive performance of fine-tuning, but may also lead to a better understanding of learned representations.