Investigating the Characteristics of a Transformer in a Few-Shot Setup: Does Freezing Layers in R o BERT a Help? (2022)

Abstract: Transformer based language models have been widely adopted by industrial and research organisations in developing machine learning applications in the presence of limited annotated data. While these models show remarkable results, their functioning in few-shot settings is still poorly understood. Hence, we perform an investigative study to understand the characteristics of such models fine-tuned in few-shot setups. Specifically, we compare the intermediate layer representations obtained from a few-shot model and a pre-trained language model. We observe that pre-trained and few-shot models show similar representations over initial layers, whereas the later layers show a stark deviation. Based on these observations, we propose to freeze the initial Transformer layers to fine-tune the model in a constrained text classification setup with K annotated data points per class, where K ranges from 8 to 64. In our experiments across six benchmark sentence classification tasks, we discover that freezing initial 50% Transformer layers not only reduces training time but also surprisingly improves Macro F1 (upto 8%) when compared to fully trainable layers in few-shot setup. We also observe that this idea of layer freezing can very well be generalized to state-of-the-art few-shot text classification techniques, like DNNC and LM-BFF, leading to significant reduction in training time while maintaining comparable performance.

## introduction the immense success of pre-trained language models (plms), such as bert @xcite has significantly fueled their adoption to several real world nlp applications. however, the massive parameterization of these models inherently assumes access to large training data to fine-tune them for specific tasks. collection of such large high quality annotated data is not only time-consuming but also a costly exercise. this gives rise to a research stream specifically focused towards develop-ing techniques that help adoption of these models in a highly constrained setting, where only a small annotated dataset is available, a setup commonly referred to as low resource setting. recent years have witnessed significant advancements in popular low-resource settings like -weak supervised learning @xcite , zero-shot learning @xcite and @xcite . despite the success of these techniques, their functioning still remains a mystery. there has been a significant amount of work done on interpretability of representations learnt by these language models in presence of large task-specific data @xcite @xcite . however, understanding the representations learnt in presence of few-shot examples is relatively less studied. hence, in this paper, we attempt to compare and contrast the characteristics of representations learnt by a bert-style language model in presence of large as well as few-shot training examples with the intention to learn better fewshot models. our work is primarily motivated from the findings of @xcite , where the authors perform a study to investigate the similarities and differences between the representations learned by plms and task-tuned models. we replicate a similar analysis on roberta-base model, where we compare the representations obtained from the plm and those obtained by fine-tuning it on sst-2 task in an oracle setup. we refer to an oracle setup, as an ideal setting where the entire plm is fine-tuned on a specific task in presence of a large training dataset. we use centered kernel alignment (cka; @xcite ) to measure similarity of representations as this is also the metric used by the authors for comparison. we observe that the representations obtained from initial layers of a fine-tuned model show high degree of similarity with those obtained from a pre-trained roberta-base model (figure 1a ). on the other hand, the representations from later layers highly deviate from the pre-trained model. this is also coherent with the observations reported by @xcite on albert @xcite model. additionally, we compare the similarity of representations obtained from plm to those obtained by fine-tuning it with k-shot examples, where we randomly sample @xmath0. we find that a similar nature of observations exists in case of k-shot model (figure 1b ), implying that most of the task-specific information is learnt in later half of the transformer layers irrespective of the size of training data used for fine-tuning. thus, based on the results from figure 1 , we observe that the representations from models finetuned in both oracle and few-shot setups capture linguistic properties similar to that of plms at the initial layers. hence, we conjecture that the initial layers can be frozen while training models in few-shot setup. we hypothesize that the reduced parameter space post such layer freezing would help learn better few-shot models. in this work, we perform a comprehensive study of the impact of freezing specific layers while finetuning language models on six popular sequence classification tasks in a constrained setup where we have access to limited dataset of k annotated examples per class, where @xmath1, 16, 32, 64}. specifically, our research contributions include: ## experimental setup based on the findings of @xcite and our experimental results in figure 1 , we investigate the impact of freezing transformer layers on training a model in a k-shot setup, where @xmath2, 16, 32, 64}. we hypothesize that freezing particular layers would significantly reduce the parameter space which would in-turn benefit the process of fine-tuning a plm specifically when we are operating in a constrained setup where we have access to only a limited number of annotated training examples. in order to test this hypothesis, we freeze the first n % of transformer layers while fine-tuning l on task d. specifically, we start with @xmath3. we continue this until @xmath4. we study this setup on both vanilla fine-tuning and state-ofthe-art (sota) few-shot techniques for fine-tuning roberta-base model on each of the benchmark datasets described in section 2.2. ## results and discussions the results obtained from experiments with vanilla fine-tuning and few-shot classification methods have been summarized in figures 2 , 3 , 4 and 5. ## prior work recent years have seen significant development in the field of language modelling using transformer @xcite based models like gpt (radford and narasimhan, 2018), bert @xcite , roberta @xcite , etc. a number of studies have been conducted to identify better techniques to fine-tune these models on nlu tasks in both oracle and few-shot settings. dodge et al. (2020), lee et al. (2020), zhang et al. (2021) focus on regularization techniques that help stabilize the fine-tuning of bert-style models. specifically, @xcite demonstrate that standard process of fine-tuning for fixed epochs is sub-optimal for bert-like models especially in few-shot setting and hence training for large epochs is required. further, @xcite show that fine-tuning on small datasets often leads to divergence during training and a simple yet effective approach like early stopping can lead to a better model selection. due to the tedious and time-consuming nature of data collection process, few-shot learning has recently started gaining popularity. @xcite propose dnnc, a nearest neighbor classification approach that uses nli-style training to predict if two inputs belong to the same class. additionally, @xcite use a prompting based approach to fine-tune a pre-trained language model in few-shot setup leading to state-of-the-art performance without introducing any new parameters. parallelly, there has been a surge in works on interpretability of language models and understanding the patterns in representations learnt by them. li et al. (2020) probe attention heads to understand certain linguistic patterns learnt by bert. @xcite design probing tasks to investigate the ability of bert-based language models in understanding properties in spoken language. @xcite discover different localizations of linguistic properties learnt by electra @xcite and xlnet @xcite . on the other hand, @xcite perform a layer wise comparison of representations learnt by pretrained and task-tuned models. while they perform an extensive analysis to compare the models in a setup where a large training data is available, the validation of these findings in the few-shot setup is largely unexplored. ## conclusions in this work, we compare the representations obtained from intermediate transformer layers of roberta-base model and task-tuned models in few-shot setup and discover that the linguistic properties learnt by pre-trained and task-tuned models at the initial layers are very similar and hence can potentially be frozen for training models in fewshot settings. we further study the impact of such freezing of transformer layers in a few-shot setting. our experimental results indicate that freezing upto initial 50% of the transformer layers surprisingly leads to performance either comparable to or better than fully trainable layers in both vanilla fine-tuning as well as sota few-shot models. we also observe that the reduced parameter space due to layer freezing leads to faster convergence which in turn leads to reduction in training time for 8-shot and 16-shot setups on both vanilla fine-tuning and sota few-shot models across tasks. specifically, for few-shot models, this observation can even be extended to 32-shot and 64-shot setups. moreover, layer freezing can be viewed as a medium to foster sustainable nlp research by reducing carbon footprint due to improvement in training efficiency. finally, our results also establish that a commonly followed practice of completely frozen encoder (100% transformer layers frozen) never helps in a few-shot setup and a proportion of transformer layers are always required to be trainable.