Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024)

Abstract: Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights.

## introduction pre-trained language models (plms) have become foundational in the transfer learning paradigm of natural language processing (nlp) @xcite @xcite . intermediate-task transfer learning aims to improve model performance further by introducing an intermediate stage of supervised training on data-rich tasks before fine-tuning the target downstream task @xcite @xcite . the paradigm has shown to be particularly useful for improving performance in resource-constrained scenarios where annotated given a target task (e.g., cb), we obtain the maximum token-wise similarity scores between the target and the source tasks for each embedding position. right: we select the source task with the highest mean of maximum similarity scores. a crucial aspect of intermediate-task transfer learning is to select beneficial tasks to transfer from. however, the costs of searching for the optimal intermediate-task, especially with the growing array of available nlp tasks and the exhaustive process of model fine-tuning @xcite , are prohibitive. research on intermediate-task selection mainly predicts task transferability using task-specific embeddings, which condense the task information of a given target task into a single vector representation. for example, some works construct task embedding from fine-tuned weights @xcite or leverage text embedding @xcite . more specifically, @xcite use sentence transformers to encode dataset examples as text embeddings. the more recent approach by @xcite constructs task embeddings from the weights of soft prompts, which have been effectively applied in large-scale studies. despite their promising results, a systematic study of the consistency of these task selection methods is still missing. specifically, it remains unclear how consistent these approaches are at predicting the best source task to transfer from. to address this gap, we perform a comprehensive evaluation of existing task selection methods in intermediate-task transfer learning. our research questions are: (1) do intermediate-task selection approaches exhibit consistent performance across downstream tasks? (2) what are the key ingredients that result in accurate transferability predictions? to answer these questions, we perform experiments across 130 intermediate and downstream task combinations derived from 13 source and 10 target tasks. our results show that intermediatetask transfer exhibits significant performance variance across tasks. comparing four representative task selection methods, we find that task embeddings based on fine-tuned weights @xcite generally outperform embedding-free and text embedding methods @xcite . however, we also observe that such task embeddings do not consistently perform well on tasks requiring highlevel reasoning abilities. exploring this further, we revisit the task embedding design and propose a new construction method based on pairwise token similarity (see figure 1 ), which yields the highest average task prediction performance of 82.5%. our main contributions are as follows: ## related work identifying a beneficial task from a broader set of source tasks is a crucial step in intermediate-task transfer learning. various studies have proposed methods to estimate task transferability based on task embeddings. a foundational approach is task2vec @xcite , which involves computing the fisher information matrix and enables to measure semantic and taxonomic relationships between tasks. in contrast, @xcite demonstrate the effectiveness of text embeddings based on sentence encoders. the landscape of task selection approaches has further evolved with the introduction of parameter-efficient fine-tuning (peft) techniques. for instance, @xcite use soft prompts to generate task embeddings, demonstrating the effectiveness of prompt-based embeddings. expanding on this, @xcite investigate other peft methods, including p-tuning @xcite , fine-tuning only bias terms @xcite . they construct task embeddings based on the fine-tuned weights. task selection based on neuron activations provides another perspective by focusing on the patterns of activations within models. @xcite propose model stimulation similarity to identify beneficial source tasks through the overlap rate of activations. more recently, @xcite introduce connectivity patterns as task embeddings, identifying task-specific patterns in deep neural networks that best represent the tasks. our work differs from previous studies by contributing a comparison of existing task selection methods in a unified setup, specifically focusing on the effectiveness and consistency of these approaches. ## background in the following, we introduce the intermediatetask transfer learning paradigm and motivate our focus on parameter-efficient fine-tuning. ## intermediate-task selection methods intermediate-task transfer can improve the performance of the target downstream task, but it is computationally infeasible to try out all possible task combinations, making choosing a beneficial source task an important problem. intermediate-task selection aims to predict task transferability and retrieve the most beneficial task from a broad set of available source tasks. this eliminates the need for exhaustive training and is more feasible in resource-constrained scenar-ios. here, we compare existing intermediate-task selection methods which can be categorized into two groups: embedding-free and embedding-based methods (see ## revisiting the construction of task embeddings despite task embeddings from fine-tuned weights demonstrating superior performance in task prediction compared to other selection methods, the effectiveness of various task embedding constructions remains underexplored. in this section, we investigate different construction methods of task embeddings. in addition to feature, we explore two more types of task embeddings as follows. ## conclusion in this work, we conduct a systematic study on intermediate-task selection across a wide range of tasks. our results show that task embeddings based on fine-tuned weights outperform random selection, data size, and text embeddings with improvements of +3.96%, +3.85%, and +2.59% in ndcg scores, underscoring the importance of a task-specific approach. nevertheless, we find that task embeddings do not excel in all scenarios, particularly in multiple choice and qa tasks. by revisiting the task embedding construction, we propose a novel method based on pairwise token similarity, which achieves the highest performance of 82.5% in task transferability prediction, suggesting that token-wise similarity is better predictive in task transferability prediction.