A dapter S hare: Task Correlation Modeling with Adapter Differentiation (2022)

Abstract: Thanks to the development of pre-trained language models, multitask learning (MTL) methods achieve a great success in natural language understanding area.However, current MTL methods pay more attention to task selection or model design to fuse as much knowledge as possible, while intrinsic task correlation is often neglected. It is important to learn sharing strategy among multiple tasks rather than sharing everything.%The MTL model is directly shared among all the tasks. %For example, in traditional MTL methods, the last classification layers or the decoder layers are manually separated. More deeply, In this paper, we propose AdapterShare, an adapter differentiation method to explicitly model the task correlation among multiple tasks. AdapterShare is automatically learned based on the gradients on tiny held-out validation data. Compared to single-task learning and fully shared MTL methods, our proposed method obtains obvious performance improvement. Compared to the existing MTL method AdapterFusion, AdapterShare achieves absolute 1.90 average points improvement on five dialogue understanding tasks and 2.33 points gain on NLU tasks.

## introduction with the development of transformer-based pretrained language models (plms), natural language understanding (nlu) has made great progress as a downstream task. there are two main ways to leverage plms in nlu tasks. one is the fine-tuning method, which updates the pre-trained language model directly on a target task. the other one is adapters @xcite , which introduces a small number of taskspecific parameters on a fixed plm. when training on the target task, only the introduced parameters are updated. compared to the fine-tuning method, adapter is memory-efficient, since the introduced parameters are much less than those of the plm. in this paper, we focus on the approach using adapters. ## problem statement as discussed, the existing multitask learning methods tend to share all parameters. it assumes that all target tasks benefit from each other. however, in practice, it can be detrimental to assume correlation in a set of tasks and simply put them together for learning @xcite . in this paper, we propose an approach to learn task correlation automatically. the task correlation indicates that all the target tasks are clustered into several task groups. the tasks in the same task group share the parameters. we maintain the task correlation map at the granularity of each transformer layer of pretrained language models. with adapters training strategy, the learning process can be formalized as: ## adaptershare in this section, we first introduce the adopted task correlation learning method in general. then we reveal the problem of existing neural differentiation algorithm and improve it in our proposed task correlation learning algorithm, adaptershare. note that in the following, all learnable parameters are adapters, while the parameters of plm are fixed. ## conclusion in this paper, we propose a robust adapter differentiation method to automatically learn task correlation in the multitask learning setting. on both encoder-decoder and encoder-only plms, our proposed method can achieve exciting performance gains compared to the separated training, completesharing and adapterfusion methods. in future work, we will try our method in the domain transfer area, which is a more general scenario than multitask learning.