Transformer Based Multi-Source Domain Adaptation (2020)

Abstract: In practical machine learning settings, the data on which a model must make predictions often come from a different distribution than the data it was trained on. Here, we investigate the problem of unsupervised multi-source domain adaptation, where a model is trained on labelled data from multiple source domains and must make predictions on a domain for which no labelled data has been seen. Prior work with CNNs and RNNs has demonstrated the benefit of mixture of experts, where the predictions of multiple domain expert classifiers are combined; as well as domain adversarial training, to induce a domain agnostic representation space. Inspired by this, we investigate how such methods can be effectively applied to large pretrained transformer models. We find that domain adversarial training has an effect on the learned representations of these models while having little effect on their performance, suggesting that large transformer-based models are already relatively robust across domains. Additionally, we show that mixture of experts leads to significant performance improvements by comparing several variants of mixing functions, including one novel metric based on attention. Finally, we demonstrate that the predictions of large pretrained transformer based domain experts are highly homogenous, making it challenging to learn effective metrics for mixing their predictions.

## introduction machine learning practitioners are often faced with the problem of evolving test data, leading to mismatches in training and test set distributions. as such, the problem of domain adaptation is of particular interest to the natural language processing community in order to build models which are robust this shift in distribution. for example, a model may be trained to predict the sentiment of product reviews for dvds, electronics, and kitchen goods, and must utilize this learned knowledge to predict the sentiment of a review about a book (figure 1 ). this paper is concerned with this setting, namely unsupervised multi-source domain adaptation. multi-source domain adaptation is a well studied problem in deep learning for natural language processing. prominent techniques are generally based on data selection strategies and representation learning. for example, a popular representation learning method is to induce domain invariant representations using unsupervised target data and domain adversarial learning @xcite . adding to this, mixture of experts techniques attempt to learn both domain specific and global shared representations and combine their predictions @xcite @xcite . these methods have been primarily studied using convolutional nets (cnns) and recurrent nets (rnns) trained from scratch, while the nlp community has recently begun to rely more and more on large pretrained transformer (lpx) models e.g. bert @xcite . to date there has been some preliminary investigation of how lpx models perform under domain shift in the single source-single target setting @xcite @xcite . what is lacking is a study into the effects of and best ways to apply classic multi-source domain adaptation techniques with lpx models, which can give insight into possible avenues for improved application of these models in settings where there is domain shift. given this, we present a study into unsupervised multi-source domain adaptation techniques for large pretrained transformer models. our main research question is: do mixture of experts and domain adversarial training offer any benefit when using lpx models? the answer to this is not immediately obvious, as such models have been shown to generalize quite well across domains and tasks while still learning representations which are not domain invariant. therefore, we experiment with four mixture of experts models, including one novel technique based on attending to different domain experts; as well as domain adversarial training with gradient reversal. surprisingly, we find that, while domain adversarial training helps the model learn more domain invariant representations, this does not always result in increased target task performance. when using mixture of experts, we see significant gains on out of domain rumour detection, and some gains on out of domain sentiment analysis. further analysis reveals that the classifiers learned by domain expert models are highly homogeneous, making it challenging to learn a better mixing function than simple averaging. ## related work our primary focus is multi-source domain adaptation with lpx models. we first review domain adaptation in general, followed by studies into domain adaptation with lpx models. ## methods this work is motivated by previous research on domain adversarial training and mixture of domain experts for domain adaptation. in this, the data consists of k source domains s and a target domain t . the source domains consist of labelled datasets d s , @xmath0, ..., k} and the target domain consists only of unlabelled data u t . the goal is to learn a classifier f , which generalizes well to t using only the labelled data from s and optionally unlabelled data from t . we consider a base network f z , @xmath1. these f z networks are initialized using lpx models, in particular distilbert @xcite . ## experiments and results we focus our experiments on text classification problems with data from multiple domains. to this end, we experiment with sentiment analysis from amazon product reviews and rumour detection from tweets. for both tasks, we perform crossvalidation on each domain, holding out a single domain for testing and training on the remaining domains, allowing a comparison of each method on how well they perform under domain shift. the code to reproduce all of the experiments in this paper can be found here foot_0 . ## discussion we now discuss our initial research questions in light of the results we obtained, and provide explanations for the observed behavior. ## conclusion in this work, we investigated the problem of multisource domain adaptation with large pretrained transformer models. both domain adversarial training and mixture of experts techniques were explored. while domain adversarial training could effectively induce more domain agnostic representations, it had a mixed effect on model performance. additionally, we demonstrated that while techniques for mixing domain experts can lead to improved performance for both sentiment analysis and rumour detection, determining a beneficial mixing of such experts is challenging. the best method we tested was a simple averaging of the domain experts, and we provided some evidence as to why this effect was observed. we find that lpx models may be better suited for datadriven techniques such as that of @xcite , which focus on inducing a better prior into the model through pretraining, as opposed to techniques which focus on learning a better posterior with architectural enhancements. we hope that this work can help inform researchers of considerations to make when using lpx models in the presence of domain shift.