Current approaches to solving classification tasks in NLP involve fine-tuning a pre-trained language model on a single target task. This paper focuses on sharing knowledge extracted not only from a pre-trained language model, but also from several source tasks in order to achieve better performance on the target task. Sequential fine-tuning and multi-task learning are two methods for sharing information, but suffer from problems such as catastrophic URLetting and difficulties in balancing multiple tasks. Additionally, multi-task learning requires simultaneous access to data used for each of the tasks, which does not allow for easy extensions to new tasks on the fly. We propose a new architecture as well as a two-stage learning algorithm that allows us to effectively share knowledge from multiple tasks while avoiding these crucial problems. In the first stage , we learn task specific parameters  that encapsulate the knowledge from each task . We then combine these learned representations in a separate combination step, termed AdapterFusion . We show that by separating the two stages, i.e., knowledge extraction and knowledge combination , the classifier can effectively exploit the representations learned from multiple tasks in a non destructive manner. We empirically evaluate our transfer learning approach on 16 diverse NLP tasks, and show that it outperforms traditional strategies such as full fine-tuning of the model as well as multi-task learning  .
 Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic URLetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks . First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information . We then combine the adapters in a separate knowledge composition step . We show that by separating the two stages, i.e., knowledge extraction and knowledge composition , the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning . Our code and adapters are available at AdapterHub.ml .
D
Current approaches to solving classification tasks in NLP involve fine-tuning a pre-trained language model on a single target task. This paper focuses on sharing knowledge extracted not only from a pre-trained language model, but also from several source tasks in order to achieve better performance on the target task.
R
two methods for sharing information, but suffer from problems such as
methods aiming to incorporate knowledge from multiple tasks; however, they suffer from
R
balancing multiple tasks. Additionally, multi-task learning requires simultaneous access to data used for each of the tasks, which does not allow for easy extensions to new tasks on the fly. We propose a new architecture as well as a two-stage
dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage
R
allows us to effectively share
leverages
R
while avoiding these crucial problems. In the first stage ,
. First, in the knowledge extraction stage
A
called adapters,
R
knowledge from each task
task-specific information
R
these learned representations
the adapters
R
combination step, termed AdapterFusion
knowledge composition step
R
combination
composition
R
non destructive
non-destructive
R
our transfer learning approach
AdapterFusion
R
NLP
NLU
R
show that it
find that it effectively combines various types of knowledge at different layers of the model. We show that our approach
D
of the model
A
. Our code and adapters are available at AdapterHub.ml