Multi-Task Supervised Pretraining for Neural Domain Adaptation (2020)

Abstract: Two prevalent transfer learning approaches are used in recent works to improve neural networks performance for domains with small amounts of annotated data: Multi-task learning which involves training the task of interest with related auxiliary tasks to exploit their underlying similarities, and Mono-task fine-tuning, where the weights of the model are initialized with the pretrained weights of a large-scale labeled source domain and then fine-tuned with labeled data of the target domain (domain of interest). In this paper, we propose a new approach which takes advantage from both approaches by learning a hierarchical model trained across multiple tasks from a source domain, and is then fine-tuned on multiple tasks of the target domain. Our experiments on four tasks applied to the social media domain show that our proposed approach leads to significant improvements on all tasks compared to both approaches.

## introduction deep learning approaches are powerful when dealing with large amounts of annotated data. however, these are only available for a few languages and domains due to the cost of the manual annotation @xcite . particularly, despite the valuable importance of social media's content for a variety of applications (e.g. public security, health monitoring, or trends highlight), this large domain is still poor in terms of annotated data. furthermore, to build systems for such applications, the machine might understand many low and high-level tasks. therefore, a model, able to recognise as many linguistic properties as possible from a given sentence, is required. many attempts have been done to exhibit the performance of nlp models in low-resource scenarios. particularly, two dominant approaches of neural transfer learning (tl) @xcite are used in the state-of-the-art (sota): 1) mono-task pretraining (mtp): sequential transfer learning (stl) @xcite , performed in two stages: pretraining on a rich source-domain on enough training examples and then, fine-tuning on the available few target-domain examples. this approach has proved to be powerful in many nlp tasks, outperforming the classic supervised learning paradigm, because it takes benefit from prelearned knowledge. and 2) multi-task learning (mtl) @xcite , that showed many benefits in several nlp tasks and applications, consists of training different tasks simultaneously, leveraging learned knowledge from related problems and resulting richer representations with higher generalisation @xcite . we introduce in this paper a novel method, that we call multi-task supervised pre-training and adaptation (mutspad), which unifies both approaches discussed above. mutspad takes benefit from both, by learning a hierarchical multi-task model trained across multiple tasks from the sourcedomain, and further fine-tuned on multiple tasks from the target-domain. hence, in addition to diverse linguistic properties learned from various supervised nlp tasks, mutspad takes advantage of the pre-learned knowledge from the high-resource source-domain. we demonstrate the effectiveness of our approach on domain adaptation from the highresource news-domain to the low-resourced tweets-domain. we carry out experiments on four nlp tasks, from the low-level part-of-speech tagging (pos) and chunking (ck) to the higher-level named entity recognition (ner) and dependency parsing (dp). mutspad exhibits significantly better performance than both tl approaches and is highly competitive compared to best sota methods. furthermore, to the best of our knowledge, there are no available common datasets containing annotations for all the above-mentioned tasks, nei-ther for the news-domain or the tweets-domain. though, many early works had highlighted the intricacy of multi-task training from heterogeneous datasets @xcite . thus, we propose to build multi-task datasets for the news and tweets domains, by unifying the aforementioned task-independent datasets. ## related work our work is related to two lines of research, sequential transfer learning and multi-task learning. in the following, we briefly present the sota of each one. then, we discuss some papers from the literature with a loosely close idea to multi-task pretraining and fine-tuning. sequential transfer learning (stl) is a tl setting performed in two stages: pretraining and adaptation. the purpose behind using stl techniques for nlp can be divided into two main research areas, "universal representations" and "domain adaptation". the former aims to build neural features transferable and beneficial to a wide range of nlp tasks and domains. e.g. elmo @xcite , bert @xcite , etc. the second aims to harness the knowledge represented in features learned on a source domain (high-resourced in most cases) to improve learning a target domain (low-resourced in most cases) @xcite . the source and the target problems may differ on the task, the language or the domain. for instance, cross-lingual adaptation has been explored for sentiment analysis @xcite and cross-domain adaptation has been applied for pos models adaptation from news to tweets domain @xcite @xcite . our research falls into the second research area, since we aim, as the last two works, to transfer the knowledge learned when training on the news-domain to improve the tweets-domain's training. multi-task learning (mtl) consists in a joint learning of related tasks and thus leverages training signals generated by different tasks @xcite . the advantage of using mtl over independent task learning has been shown in many nlp tasks and applications @xcite . the processing (training) order of examples from different tasks (datasets), also called scheduling, is particularly studied in the literature. this could be implicit or explicit @xcite . the formal include, for instance, affecting different learning rates for task-specific parameters. while the second, modify the importance of each task statically or dynamically. e.g. kiperwasser and ballesteros (2018) proposed variable schedules that increasingly favour the principal task over batches and @xcite proposed adaptive schedules that vary according to the validation performance of each task during training. multi-task pretraining and fine-tuning: multitask pretraining has been especially explored for learning universal representations @xcite . multi-task fine-tuning was recently explored to fine-tune bert pretrained model in a multi-task fashion on multiple tasks @xcite . furthermore, in term of using multi-task features for domain adaptation, @xcite showed the benefit of multi-task learning for domain adaptation from news-domain to weblogs-domain for ck task, when disposing ck's supervision only for the source-domain, and lower-level pos supervision for the target-domain. finally, in terms of unifying multi-task learning and fine-tuning, kiperwasser and ballesteros (2018) proposed to improve machine translation with the help of pos and dep tasks by scheduling tasks during training, starting with multi-tasking of the principal task with auxiliary lower-level tasks (pos and dep), and as the training graduates, the model trains only to the main task. however, to the best of our knowledge, performing pretraining and fine-tuning on multitask models for domain adaptation has not been explored in the literature. ## our approach in low-resources scenarios, two approaches are commonly used to alleviate the lack of training data: the first approach is known to work very well since a while, and yielded impressive results in recent years but the last approach seems to struggle as it still faces the problem of annotated data scarcity. in this paper, to make sure that we learn useful features that are relevant for the tasks of our interest, we propose an approach that combines pretraining and multi-task learning, and thus takes benefits from the rich source-domain, and especially all its available annotated data and tasks. we called our method multi-task supervised pre-training and adaptation (mutspad). this is described in details in sec. 4.1, before we describe (in sec.4.2) how we alleviated the problem of datasets heterogeneity in multi-task learning (i.e., only mono-task labelled datasets are available). ## conclusion in this research, we have proposed mutspad, a new approach based on transfer learning (tl) for domain adaptation with three main contributions: 1) consolidating two tl's approaches, sequential transfer learning and multi-task learning, by pretraining on resource-rich domain and finetuning on low-resourced domain in a multi-task fashion; 2) unifying independent datasets to overcome the intricacy of multi-task training from different datasets; and 3) conducting a set of individual units ablation, refining our understanding on how individual neurons lower-level tasks impact high-level tasks. we showed through empirical results on domain adaptation from news to tweets that the proposed method mutspad allows a simultaneous benefit from similarities between domains and tasks, yielding better transfer learning performances on four nlp tasks, and outperforming the state-of-the-art on dependency parsing task. this study leaves several important open directions for future work. first, we should explore soft multi-task architectures. second, we expect to explore the combination of supervised and unsupervised multi-tasking. we also plan to explore the benefit of mutspad's learned representations for higher-level nlp applications such as machine translation and sentiment analysis.