Revisiting Parameter-Efficient Tuning: Are We Really There Yet? (2022)

Abstract: Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of parameters comparing to full model finetuning, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investigation into the training and evaluation of them. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competitive performance while finetuning remains to be the best-performing method in medium- and high-resource settings. We delve deeper into the cause of the instability and observed that the number of trainable parameters and training iterations are two main factors: reducing trainable parameters and prolonging training iterations may lead to higher stability in PETuning methods.

## introduction pretrained language models (plms) such as bert @xcite and roberta @xcite have orchestrated tremendous progress in nlp in the past few years, achieving state of the art on a large variety of benchmarks such as glue @xcite and superglue @xcite . most successful applications of plms follow the pretraining-and-finetuning transfer learning paradigm @xcite , where plms are used as backbones to be combined with additional parameters and finetuned on downstream tasks in an end-to-end manner. whilst being simple and effective, such paradigm requires task-specific tuning of the full model that consists of hundreds of millions @xcite , or even billions @xcite @xcite of parameters for each task, which is time-consuming and resource-intensive. to avoid full model finetuning, there has been a surge of studies on parameter-efficient tuning (petuning) methods, which aim to tune the plms by adjusting lightweight trainable parameters while keeping most pretrained parameters frozen. various ways have been used in these petuning methods to introduce the lightweight trainable parameters. adapter @xcite ) is one of these that injects a small portion of model-level parameters within each trans-former @xcite layer of the pretrained language model. prompt-tuning @xcite @xcite is another class of methods that introduce trainable continuous embeddings into the original sequences of input token embeddings to augment the plms on the feature level. diff-pruning @xcite learns and updates additional sparse diff-vector for all pretrained parameters, and lora @xcite learns low-rank matrices to approximate the updated matrices, both of which update the plms on the parameter level. moreover, bitfit @xcite partially tunes the bias terms of plms, without even introducing any new parameters. more details for these methods can be seen in appx. §a. given the various exciting progresses of petuning methods that all seem to demonstrate their competitive performance with higher training efficiency, the idea that petuning could be a new general paradigm in place of full finetuning for transfer learning in nlp becomes never more tempting @xcite . we, however, argue that current evidences are insufficient to support the complete overthrow of full finetuning. first, we point out that the current evaluation strategy, i.e., the development set is used for both early stopping and reporting results, used in a number of studies for petuning @xcite @xcite does not provide fair model comparisons. this essentially causes data leakage that results in misleading conclusions ( §2). second, statistical significance is rarely reported when comparing petuning methods. this is an especially crucial issue as we show that the finetuning and petuning processes are inherently unstable due to various randomness, such as weight initialization and training data order ( §3.3). to fairly compare these tuning strategies, this study conducts a comprehensive re-examination on the effectiveness of petuning methods. our main contributions are: 1) we conduct controlled experiments ( §2) and reveal the fundamental flaw of the current evaluation scheme (i.e., its failure to assess generalisation) and how that leads to misinterpretations of the progress in the field. 2) we offer a more reliable practice for model selection that is not prone to overfitting. 3) we revisit the performance of petuning in comparison with finetuning across tasks with various, and have reached very different conclusions on different data scales. ## experiments with fair evaluation in this section, we use a fairer evaluation protocol that strictly separates dev and test sets. based on this protocol, we conduct extensive experiments to investigate the effectiveness of petuning methods (concluded in figure 1 ). first, we experiment over a wide range of tasks under various levels of resource abundance to comprehensively compare the performance of petuning with finetuning ( §3.2). further, we provide in-depth analyses for the instability of petuning methods, investigating the possible causes and provide practical suggestions of using petuning methods ( §3.3). ## conclusion this work conducted a rigorous re-examination on the current parameter-efficient tuning (petun-ing) methods. we demonstrated that performing early stopping and evaluation on the same dataset (a common practice used in many past studies) could lead to unreliable conclusions. this issue is more pronounced when accompanied by the instability nature of petuning, leading to inflated results and overly optimistic estimates of petuning approaches. we re-evaluated these petuning methods on the performance and stability aspects on a rigorous evaluation protocol that strictly separates validation and test sets. by conducting a set of fine-grained comparisons between petuning and finetuning, we found that petuning methods are not consistently competitive with finetuning. namely, prefix tuning performs poorly across tasks and most petuning methods perform worse than finetuning on higher-resource settings. by systematically investigating the instability of petuning methods, we found that models' instability is sensitive to both weight initialisation and training data order. we identify two major factors behind such instability: 1) models with fewer parameters are more stable within each petuning method class; 2) more training iterations can usually reduce instability. our overall re-examination conclude that finetuning still cannot be fully replaced by petuning so far, and there are many key challenges for petuning in terms of both performance and instability, which need to be addressed in future work.