Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer (2023)

Abstract: Recent studies show that large-scale pre-trained language models could be efficaciously adapted to particular tasks in a parameter-efficient manner. The trained lightweight set of parameters, such as adapters, can be easily stored and shared as a capability equipped with the corresponding models. Owning many lightweight parameters, we focus on transferring them between tasks to acquire an improvement in performance of new tasks, the key point of which is to obtain the similarity between tasks. In this paper, we explore 5 parameter-efficient weight ensembling methods to achieve such transferability and verify the effectiveness of them. These methods extract the information of datasets and trained lightweight parameters from different perspectives to obtain the similarity between tasks, and weight the existing lightweight parameters according to the comparability to acquire a suitable module for the initialization of new tasks. We apply them to three parameter-efficient tuning methods and test them on a wide set of downstream tasks. Experimental results show that our methods show an improvement of 5%~8% over baselines and could largely facilitate task-level knowledge transfer.

## introduction increasingly large pre-trained language models (ptms) @xcite @xcite have yielded exceptional performances on a variety of tasks but also suffer from prohibitive adaptation costs with full parameter fine-tuning. it is not a feasible choice to fine-tune all parameters of a colossal model for each specific downstream task and produce a corresponding instance at the same size. to overcome this obstacle, a branch of research, namely parameter-efficient tuning, has * equal contributions † corresponding authors been actively developed and explored @xcite @xcite @xcite . it demonstrates that only optimizing a tiny portion of parameters and keeping the ptm frozen could achieve on-par performance with full parameter fine-tuning on many tasks. after training, the set of updated parameters is lightweight and portable for storing and sharing. although the specific structures of these parameters may be different, we treat them in a unified perspective and call them lightweight objects. once lightweight objects are trained, they can be adapted to specific datasets conditioned on a large-scale ptm and be placed aside for storage in a space-efficient manner. due to its lightweight nature, it is pragmatic to build a platform to store and share such lightweight objects for various scenarios @xcite . however, the maneuverability of the platform for storing and sharing lightweight objects is still not fully exploited. in the current paradigm, one could directly access and utilize lightweight objects trained on existing datasets but face hindrances in utilizing the knowledge of these objects for new datasets @xcite . such existing lightweight objects can be a valuable resource, as the knowledge contained in them has the probability of transferring to similar tasks. in this paper, we focus on the transferring of lightweight objects and aim to leverage them to boost the performance of new datasets. we assume that more similar tasks can share more knowledge, so the key point is to acquire the similarity between existing lightweight objects and new tasks. specifically, we first assess 3 straightforward approaches to facilitate parameter-efficient tuning on new datasets. observing unsatisfactory results on such approaches, we develop a parameter-efficient weight ensembling framework that could produce a set of parameters according to new datasets and existing lightweight objects. under the framework, we explore 5 ## investigated methods we consider a scenario where pre-trained language models m are frozen. and for downstream tasks {t 1 , t 2 , ..., t n }, the associated lightweight modules o * = {o 1 , o 2 , ..., o n } are produced via parameter-efficient fine-tuning (e.g., lora (hu et al., 2021), adapter @xcite ). given a new task t new with a few examples, our goal is to explore the best approach to utilize pre-existing lightweight objects to cultivate the best-performing lightweight object o new for the initialization of t new . we investigate 9 strategies to transfer lightweight objects across tasks, including 3 baselines and 5 particular methods under our parameter-efficient weight ensembling framework. ## experiments in this section, we apply the aforementioned approaches in different scenarios for experimental comparisons and analysis. ## conclusion