Dynamic Task Vector Grouping for Efficient Multi-Task Prompt Tuning (2025)

Abstract: Multi-task prompt tuning utilizes multiple high-resource source tasks to improve performance on low-source target tasks. Existing approaches transfer the soft prompt trained by combining all source tasks or a single “high-similar” source task one-time-only. However, we find that the optimal transfer performance often comes from a combination of source tasks, which is neither one nor all. Further, we find that the similarity between source and target tasks also changes dynamically during fine-tuning after transfering, making similarity calculation in the initiation stage inadequate. To address these issues, we propose a method called Dynamic Task Vector Grouping (DTVG), whose core ideas contain (1) measuring the task similarity with task vectors instead of soft prompt, (2) grouping the optimal source task combination based on two metrics: target similarity and knowledge consistency ; (3) dynamically updating the combination in each iteration step. Extensive experiments on the 26 NLP datasets under different settings demonstrate that DTVG effectively groups similar source tasks while reducing negative transfer, achieving the start-of-art performance.

## introduction full parameter fine-tuning (ft) of large pre-trained language models (plms) has shown significant success in addressing various natural language processing (nlp) tasks. however, the conventional fine-tuning paradigm requires substantial memory and computational resources. recently, parameter efficient fine-tuning (peft) @xcite @xcite @xcite aims to achieve comparable results of ft by updating a significantly small set of the model parameters. we incrementally add source tasks with a random order and train soft prompt by examples-proportional mixing @xcite . in the bottom part, we calculate the cosine similarity between the average pooled representations of the prompt tokens @xcite . we initialize the rte soft prompt using the source task's soft prompt with the highest similarity. the legend marker denotes the source task with the highest similarity, which shifts from qnli to mnli during fine-tuning. soft prompt tuning (pt) @xcite , as an effective peft method, achieves a trade-off between effectiveness and efficiency. during training, a series of learnable soft prompt vectors prepended to the input are updated while the original plms are frozen. unlike methods such as lora @xcite and adapter @xcite , pt is independent of the model architecture and can be applied to various models without modification. although promising, the existing study @xcite demonstrates pt still underperforms compared to ft, particularly in the case of low-resource tasks. an additional issue with pt is sensitivity to the initialization and needs longer tuning for converge @xcite . recent works @xcite @xcite address the above limitations by transferring soft prompt from high-resource source task to low-resource target task. specifically, they initialize the soft prompt for the target task by either (1) learning a common soft prompt across all source tasks or (2) learning a soft prompt for each source task and selecting one with the task similarity. subsequently, the soft prompt is tuned exclusively using limited training samples from the target task. these transfer approaches effectively maintain the parameters efficiency of soft prompts and demonstrate superior performance compared to vanilla prompt tuning. despite substantial progress, we challenge the rationality of some straightforward ideas in existing approaches. we first check whether existing methods achieve optimal performance. in the upper part of figure 1 , we observe that a subset of source tasks achieves the best transfer performance, neither all source tasks nor a source task. additionally, @xcite demonstrates mnli, qnli, and qqp positively transfer to the rte dataset, while we find that their gradual addition does not yield a consistent monotonic improvement due to the potential conflicts among source tasks. these observations revealed that we should find a group of source tasks for each target task and consider potential conflicts between source tasks besides the similarity to target tasks. further, we check whether "the most similar source task" will change in the tuning stage of the target task. we study a single-task version of spot @xcite , which transfers the soft prompt from a source task to initiate the target task via similarity measure between their learned soft prompt. in the bottom part of figure 1 , we find that "the most similar source task" of rte shifts from qnli to mnli over time. recall that the lowresource characteristics of the target task hinder sufficient convergence of soft prompt; therefore, it is unsurprising that we cannot select the truly most similar task with an unconverted soft prompt of the target task. this observation suggests that dynamically updating the selected source task during the target task's fine-tuning may enhance the sustainable acquisition of knowledge. motivated by these valuable empirical observations, we propose a method called dynamic task vector grouping (dtvg). specifically, we first introduce a novel task similarity metric, the dot product between task prompt vectors (tpv), which steadily achieves a better transfer performance than the current metric, the cosine similarity between soft prompts. based on this metric, we introduce a source task grouping method to select the transfer source task group for each target task with two metrics, including target similarity and knowledge consistency. then, a multi-task merging method is used to weighted sum the task vectors from the target task and the selected source tasks, synthesizing the initialization soft prompt for the target task. during the fine-tuning stage of the target task, we track the task similarity changes and dynamically update the source task group, which will effectively improve transfer performance. in summary, our major contributions are to: although the pt method has shown great success in various nlp tasks, it still faces the lowsource challenge: too few training samples prevent the soft prompts from converging, which can result in huge performance differences under different soft prompt initializations. ## conclusion in this paper, we present dtvg, a novel approach for addressing potential negative transfer in multitask prompt tuning based on task prompt vectors. compared to vanilla transfer of the soft prompt from all source tasks, we dynamically group a subset of source tasks and merge their task prompt vectors to avoid an unrelated source task inducing performance degradation of the target task. extensive $ffxudf\ n n n n n n 3urpsw7xqlqj dtvg2xuv 9dolgdwlrq3huirupdqfhrq57( 01/,667 01/,41/, 01/,443 figure 7 : validation performance on rte with source task grouping. the source tasks are arranged in each patch legend from left to right, ordered by their similarity to the target task, from highest to lowest.