H yper PELT : Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023)

Abstract: With the scale and capacity of pretrained models growing rapidly, parameter-efficient language model tuning has emerged as a popular paradigm for solving various NLP and Vision-and-Language (V&L) tasks. In this paper, we design a unified parameter-efficient multitask learning framework that works effectively on both NLP and V&L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings and visual modality as input, and outputs weights for different modules in a pretrained language model, such as the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning.). Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V&L tasks confirm the effectiveness of our framework.

## introduction pretraining and fine-tuning are now the prevalent paradigm in natural language processing, yielding state-of-the-art performances on a variety of tasks @xcite . with pre-trained language models (plms) growing rapidly in size, it becomes increasingly infeasible to perform conventional fine-tuning on the entire model parameters. there has recently been one line of research on parameter-efficient language model tuning (pelt) @xcite @xcite . they only update a set of extra trainable task-specific parameters that are newly introduced to plms. although the number of new parameters is much fewer than the * work is done at the internship of noah's ark lab, huawei technologies. empirical results on 8 tasks of glue benchmark show that hyperpelt achieves superior performances (87.09 vs. 86.53) with a tenth of the parameters (0.24% vs. 2.96%) when compared to state-of-the-art alternatives. study on the few-shot transfer learning indicates that hyperpelt is more stable and efficient than alternatives. it confirms the effectiveness of our unified parameter-efficient multitask learning framework. what's more, we evaluate our framework on v&l multi-tasks (4 tasks). results show the promising performance of our novel fusion method on extending v&l ability on top of plms via hypernetworks. in summary, we make the following contributions: (1) propose a unified parameter-efficient multitask learning framework that is able to take multi-task and multi-modality information as input, and generate weights for tuning different taskspecific modules of plms; (2) present a novel perspective of using hypernetworks to achieve the parameter-efficient multimodal fusion on top of plms; (3) design various experiments to comprehensively demonstrate the effectiveness of our proposed framework in multi-task learning and fewshot domain transfer scenarios. ## related work existing research has explored a large amount of methods on parameter-efficient tuning, such as the widely used adapter-tuning @xcite , prefix-tuning (li and liang, 2021) and the mixed methods @xcite . however, it is time & space-consuming to deal with a set of tasks in multi-task learning if we simply update and save separate replicas of model parameters per single task. in this work, we explore a hypernetwork-based multi-task learning framework to generate weights for different pelt modules. besides, there has been a series of recent work @xcite @xcite to equip a language model with the ability of handling visual input with a small number of trainable modules and parameters. different from existing work, we propose a novel perspective of multimodal fusion via extending the proposed parameter-efficient multitask learning framework. we further review recent research on parameter-efficient tuning for pure language and v&l tasks, as well as the corresponding work for multi-task learning in appendix a. ## proposed method we target a general multi-task learning problem, which is formulated in appendix b. in this section, we describe the hyper-embedding i for hypernetworks to generate weights ∆θ and which modules of plms to insert these weights to achieve pelt. in our methods, the hyper-embedding i consists of two: task-specific hyper-embedding i τ , and visualspecific hyper-embedding i v . we will mostly introduce the hyper-embedding i τ , and i v is used in a similar parallel manner. a simple linear projection layer is employed as the hypernetwork, for example, h τ p (.) and h v p (.) are used for prefix-tuning, while h τ a (.) and h v a (.) are for adapter-tuning as shown in figure 1 . the hypernetwork takes the hyper-embedding i as input, and outputs weights for multiple modules of plms. ## results and analysis we conduct a series of experiments to verify the effectiveness of our proposed framework compared to existing ones. ## discussion and conclusion in this paper, we propose a unified parameterefficient tuning framework for multitasks. on the one hand, we use the hypernetwork to reduce the scale of trainable parameters of existing adaptertuning and prefix-tuning modules. on the other hand, for the v&l tasks, we directly integrate the image features into the prefix vectors as well as adapters, which further reduces the number of trainable parameters for processing visual input. extensive experiments on pure language and v&l tasks demonstrate the superiority of our proposed framework in both multi-tasking and few-shot settings. in the future, we plan to explore more combination of methods across tuning task-specific and visualspecific parameters for different modules of plms.