Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks (2022)

Abstract: Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications. Recent studies have explored parameter-efficient PLM tuning, which only updates a small amount of task-specific parameters while achieving both high efficiency and comparable performance against standard fine-tuning. However, all these methods ignore the inefficiency problem caused by the task-specific output layers, which is inflexible for us to re-use PLMs and introduces non-negligible parameters. In this work, we focus on the text classification task and propose plugin-tuning, a framework that further improves the efficiency of existing parameter-efficient methods with a unified classifier. Specifically, we re-formulate both token and sentence classification tasks into a unified language modeling task, and map label spaces of different tasks into the same vocabulary space. In this way, we can directly re-use the language modeling heads of PLMs, avoiding introducing extra parameters for different tasks. We conduct experiments on six classification benchmarks. The experimental results show that plugin-tuning can achieve comparable performance against fine-tuned PLMs, while further saving around 50% parameters on top of other parameter-efficient methods.

## introduction in many industrial applications about natural language processing, it becomes a de-facto paradigm that we first pre-train large-scale language models (plm) @xcite @xcite on external corpora and then fine-tune them on target tasks. however, each finetuned plm is generally applicable to only one task. as the number of applications increases, deploying independent instances of fine-tuned plms for different tasks significantly increases the computation and storage costs. for instance, @xcite ) contains 175b parameters, which makes it almost impossible to fine-tune and deploy gpt-3 for target tasks. even if large-scale fine-tuned plms are available, they are also not conducive to community distribution and sharing. recently, a new branch of researches named parameter-efficient tuning @xcite @xcite have received much attention in the nlp community. compared to standard fine-tuning, these methods fine-tune only a small portion of the model parameters while keeping most of the plm parameters frozen @xcite . the minimal trainable parameters not only remarkably promote the deployment and storage efficiency when adapting plms, but also make it feasible to train large-scale plms such as gpt-3. additionally, recent works validate that the parameter-efficient methods can achieve comparable performance with full-parameters finetuning in a wide range of nlp applications @xcite @xcite . however, all the existing parameter-efficient methods ignore the inefficiency problem caused by the specific output layers for different tasks, which results in two problems. first, utilizing different task paradigms and retaining different output layers is inflexible and hinders the community model distribution and sharing. second, the parameters introduced by the task-specific output layers are non-negligible especially under the parameterefficient scenarios. these two essential issues limit parameter-efficient tuning from being further efficient. in this work, we propose plugin tuning, a framework to further improve all the parameterefficient methods in classification tasks. we unify all the classification tasks into a same paradigm, thus different tasks can be performed with a unified classifier. meanwhile, no trainable parameter is required for training the classifier. specifically, we re-formulate different tasks into a same language modeling task, and directly reuse the language model heads of plms for classification. the label spaces of different tasks are mapped to task-specific label words, all belonging to a unified vocabulary space. to select the proper label words, we propose a principled algorithm that is applicable to both token and sentence classification tasks. in this way, the efficiency of all the parameter-efficient methods can be largely promoted. additionally, the unified task paradigm provides a new way to perform classification tasks with generative plms such as gpt-3, which is unexplored in existing parameterefficient methods. our codes are publicly available at github foot_0 . our contributions can be summarized as follows: ## our method in this work, we propose plugin-tuning, a unified framework to improve parameter-efficient tuning. our method is based on parameter-efficient tuning but avoids the problems caused by task-specific output layers. an overview of plugin-tuning is shown in figure 2 . the plugin here refer to the trainable parameters required for parameterefficient method. we inject the plugin to the frozen plm, and the plugin is applied to a specific position to control the plm, as shown in @xcite . the plm will predict the label word in label-related position of sentence, and the real label can be obtained by label word mapping. any parameterefficient method can serve as our plugin. we only optimize the lightweight plugin during training while keeping other parameters frozen. the details of the unified classifier are shown in the following subsections. ## experiment we verify the effectiveness of the plugin-tuning method on a large number of tasks including both token and sentence classification tasks, and select representative parameter-efficient methods to demonstrate that plugin-tuning is compatible with arbitrary parameter-efficient methods. in addition, we also simulate real scenarios and counted the deployment time to further verify the efficiency of our method in deployment. finally, we provide a comprehensive analysis of the unified classifier, which is the main module of the plugin-tuning. ## conclusion in this work, we propose plugin-tuning, a unified framework to improve parameter-efficient tuning for both token and sentence classification tasks. we use a unified classifier to handle different classification tasks by re-formulating them to a unified language modeling task, where no trainable parameter is required for training the classifier. to select the proper label words, we also propose a principled algorithm that is applicable to both token and sentence classification tasks. in this way, the efficiency of all the parameter-efficient methods can be largely promoted. the experiments show that our method achieves comparable performance against fine-tuned plms while further saving up to 49% parameters on top of other parameter-efficient methods. future directions might be applying the unifying strategy into other tasks like text matching that require specific label spaces.