Parameter-Efficient Tuning Makes a Good Classification Head (2022)

Abstract: In recent years, pretrained models revolutionized the paradigm of natural language understanding (NLU), where we append a randomly initialized classification head after the pretrained backbone, e.g. BERT, and finetune the whole model. As the pretrained backbone makes a major contribution to the improvement, we naturally expect a good pretrained classification head can also benefit the training. However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining ineffective. In this paper, we find that parameter-efficient tuning makes a good classification head, with which we can simply replace the randomly initialized heads for a stable performance gain. Our experiments demonstrate that the classification head jointly pretrained with parameter-efficient tuning consistently improves the performance on 9 tasks in GLUE and SuperGLUE.

## introduction fine-tuning is the most prevalent paradigm to leverage pretrained language models for the best performance on specific tasks @xcite . usually, a task-oriented classification head is grafted onto the final layer of the pretrained backbone, mostly transformers @xcite , and then the whole model is trained for the downstream task. compared with training a large transformer from scratch, the pretrained backbone has already learned to extract features about grammar, semantics and high-level understanding, making it easy to adapt the model for nlp tasks. although the good initial weights of the pretrained backbone are the key factor of the effectiveness, the initial weights of the classification heads are, however, largely under-explored. to the best of our knowledge, the usage of randomly initialized classification heads is still the overwhelmingly dominant choice in nlp. recently, lp-ft @xcite finds that on many computer vision benchmarks, the performance can be promoted, especially for out-of-distribution (ood) data, if we first only finetune the linear classification head (probe) with the pretrained backbone frozen, and then finetune the whole model. does this kind of head-first finetuning technique also work in nlp? furthermore, since the principle is basically to reduce the change of features during finetuning (figure 1 ), is it possible to upgrade lp-ft to a general method to improve generalization, instead of only ood setting? in this paper, we give a positive answer via parameter-efficient tuning. to answer these questions, we need to first understand why a well-initialized head helps. lp-ft explains that with a randomly initialized head, the extracted features from the backbone change greatly during fitting the in-distribution (id) samples in the training set, but the extracted features for ood samples change very little. finally, the training makes the features inconsistent for id and ood samples. a good classification head from linear probing can reduce the intensity of change in the extracted features during finetuning, and thus keep the feature consistency. since the overfitting-like explanation above only relates to the training samples, it should still be reasonable if we replace the "id/ood samples" by "training/test samples" in the explanation. however, both our and the original lp-ft experiments suggest no significant improvement for the (id) downstream task itself, why is that? in our opinion, the inconsistency is attributed to that the head-only tuning in lp-ft cannot obtain a good enough classification head if the pretrained task is greatly different from the downstream task. the head-only tuning usually has much worse performance than fully funetuning, so that even with the pretrained head, in the fully finetuning stage the weights in the backbone still need to change greatly to a totally different local minima for better performance. in this paper, we define the criterion of good classification heads as those with which the pretrained backbone can be optimized to a near-optimal point with little change. the recently rising parameterefficient tuning methods @xcite @xcite @xcite aim to approach the performance of fully finetuning by only changing a very small part of parameters, and thus fulfil our desires. our experiments show that the parameterefficient tuning methods, such as prefix-tuning (li and liang, 2021), bitfit @xcite and @xcite indeed make good classification heads. initialized by these good clas-sification heads, the performances on 9 tasks of glue @xcite and superglue @xcite are consistently better than or equal to that of direct finetuning or lp-ft. ## conclusion finetuning the pretrained model using a randomly initialized classification head in a downstream task may result in the model output feature deviating too far. we propose efficient head finetuning (eh-ft), an efficient head pretraining strategy using parameter-efficient tuning and only introduce a little extra time and space while improving the model with a stable performance gain in different tasks. eh-ft can make a good initialization of head which can guide model to a local minimum close to the pretrained point, alleviating the catastrophic forgetting and overfitting of large-scale pretrained models. furthermore, this method can be applied to any pretrained model, as long as it only needs a classification head for downstream tasks.