Connectivity Patterns are Task Embeddings (2023)

Abstract: Task embeddings are task-specific vectors designed to construct a semantic space of tasks, which can be used to predict the most transferable source task for a given target task via the similarity between task embeddings. However, existing methods use optimized parameters and representations as task embeddings, resulting in substantial computational complexity and storage requirements. In this work, we draw inspiration from the operating mechanism of deep neural networks (DNNs) and biological brains, where neuronal activations are sparse and task-specific, and we use the connectivity patterns of neurons as a unique identifier associated with the task. The proposed method learns to assign importance masks for sub-structures of DNNs, and accordingly indicate the task-specific connectivity patterns. In addition to the storage advantages brought by the binary masking mechanism and structured sparsity, the early-bird nature of the sparse optimization process can deliver an efficient computation advantage. Experiments show that our method consistently outperforms other baselines in predicting inter-task transferability across data regimes and transfer settings, while keeping high efficiency in computation and storage.

## introduction with the rapid development and excellent performance of large pre-trained language models (plms), the most prevalent paradigm in natural language processing (nlp) has become pretraining then fine-tuning @xcite @xcite @xcite . extending upon the two-step training procedure, previous works show that intermediate-task transfer, i.e., fine-tuning the model on an intermediate source task before the target task, can yield further gains @xcite , 2018; @xcite . nevertheless, the improvement by intermediate-task transfer heavily relies on the selection of a proper intermediate task because some source tasks lead to performance degradation @xcite . one straightforward approach is to enumerate every possible (source, target) task combination, but it is extremely expensive. therefore, recent works explore methods to predict inter-task transferability accurately with high efficiency. the current state-of-the-art (sota) works are established on task embeddings, (i.e., leveraging a single vector to represent a task). they predict inter-task transferability by computing the similarity between task embeddings. task2vec @xcite develops task embeddings based on the fisher information matrix while requiring fine-tuning the full model and consuming a large amount of storage @xcite . recently, researchers propose that the efficiently tuned parameters like prompts @xcite and lora @xcite encode rich information for a task and thus can serve as task embeddings @xcite @xcite . however, these tuned parameters are sensitive to model initialization and stochasticity @xcite , and optimizing these parameters consumes significantly more computational resources than traditional finetuning @xcite . different from them, we draw inspiration from the shared working mechanisms of dnns and biological brains to develop high-quality task embeddings. we start by considering which parts of knowledge within the model are being utilized for a given task. typically, recent works in sparse optimization and model pruning have shown that sub-structures (e.g., neurons, attention heads, channels, and layers) from different parts of the model exhibit specialization in distinct knowledge and possess varying degrees of importance for a particular task @xcite @xcite @xcite . these are consistent with the findings in neuroscience that activities of neurons and connectivities in biological brains are sparse @xcite @xcite and task-specific @xcite @xcite . the aforementioned remarkable findings motivate us to use task-specific connectivity patterns in dnns to represent tasks. in this work, we propose a novel task embedding, namely connectivity patterns as task embedding (copate), and apply it to predict the inter-task transferability, as illustrated in figure 1 . our key insight is that in over-parameterized dnns, there exist connectivity patterns (i.e., the structures of subnetworks) that are functional for one certain task, and can capture high-density task-specific information. concretely, we assign importance masks to attention heads and intermediate neurons of plms, jointly train the masks and the model, and extract task embeddings according to the learned masks. our method has two strengths in efficiency: 1) it is computation-friendly as we extract connectivity patterns early in the training; 2) it is storage-friendly because our embedding granularity is coarse-grained, and copate can be represented by a binary mask. experiments show that compared to other approaches, copate has superior inter-task prediction capability across data regimes and transfer settings. our codes are available at github 1 . in this section, we demonstrate the framework to identify task-specific connectivity patterns. we represent the task-specific connectivity patterns via the structure of essential subnetworks found by sparse optimizing and pruning techniques @xcite @xcite , including the searching stage (sec 2.1) and the extracting stage (sec 2.2). ## copate: connectivity patterns as task embedding in this section, we first show how we generate task embeddings with task-specific connectivity patterns at hand (sec 3.1). next we provide empirical evidence for the appropriateness of using the obtained task embeddings to predict inter-task transferability in sec. 3.2. ## predicting task transferability in this section, we perform thorough experiments to empirically demonstrate the capability of copate in predicting inter-task transferability. ## related work predicting beneficial intermediate tasks it has been shown that intermediate-task transfer can deliver performance gains for many target tasks @xcite @xcite , but improper intermediate tasks can result in negative transfer results @xcite . hence, researchers try to accurately identify the most beneficial source task based on metadata or extracted representations of tasks (alonso and plank, 2017; @xcite . recent works represent tasks with embeddings that are generated from data representations @xcite , model weight information @xcite , and efficiently tuned parameters @xcite @xcite . different from them, we start from a model architecture perspective and use connectivity patterns to represent tasks. techniques to obtain sparse subnetworks researchers have explored a variety of techniques to obtain sparse networks by removing sub-structures like weights @xcite @xcite , channels @xcite @xcite , attention heads @xcite @xcite and layers @xcite . these approaches first identify unimportant sub-structures and subsequently remove them. with the increasing size of plms, sparse subnetworks have become increasingly important for efficient deployment and inference in nlp, lead-ing to a proliferation of related research @xcite @xcite . our proposed method, which uses connectivity patterns as task embeddings, is orthogonal to these existing techniques. ## conclusion in this work, we propose copate, a novel task embedding that represents tasks with sparse connectivity patterns, and develop a method to get such embeddings. comprehensive experiments show that the proposed method outperforms other competitive approaches in predicting inter-task transferability while achieving efficiency in both computation and storage. we hope that our work may motivate future work in introducing connectivity patterns as task embeddings to fields like meta learning, multi-task learning, and model interpretability.