Neural Unsupervised Domain Adaptation in NLP — A Survey (2020)

Abstract: Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks. In contrast, learning from unlabeled data, especially under domain shift, remains a challenge. Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data. This is a more challenging yet a more widely applicable setup. We outline methods, from early traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention. Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future NLP.

## introduction deep learning has undoubtedly pushed the frontier in natural language processing (nlp). particularly large pre-trained language models have improved results for a wide range of nlp applications. however, the lack of portability of nlp models to new conditions remains a central issue in nlp. for many target applications, labeled data is lacking (y scarcity), and even for pre-training general models data might be scarce (x scarcity). this makes it even more pressing to revisit a particular type of transfer learning, namely domain adaptation (da). a default assumption in many machine learning algorithms is that the training and test sets follow the same underlying distribution. when these distributions do not match, we face a dataset shift @xcite -in nlp typically referred to as a domain shift. in this setup, the target domain and the source training data differ, they are not sampled from the same underlying distribution. consequently, performance drops on the target, which undermines the ability of models to truly generalize into the wild. domain adaptation is closely tied to a fundamental bigger open issue in machine learning: generalization beyond the training distribution. ultimately, intelligent systems should be able to adapt and robustly handle any test distribution, without having seen any data from it. this is the broader need for out-of-distribution generalization @xcite , and a more challenging setup targeted at handling unknown domains @xcite . work on domain adaptation focused largely on supervised domain adaptation @xcite . in such a classic supervised da setup, a small amount of labeled target domain data is available, along with some larger amount of labeled source domain data. the task is to adapt from the source to the specific target domain in light of limited target domain data. however, annotation is a substantial timerequiring and costly manual effort. while annotation directly mitigates the lack of labeled data, it does not easily scale to new application targets. in contrast, da methods aim to shift the ability of models from the traditional interpolation of similar examples to models that extrapolate to examples outside the original training distribution @xcite . unsupervised domain adaptation (uda) mitigates the domain shift issue by learning only from unlabeled target data, which is typically available for both source and target domain(s). uda fits the classical real-world scenario better, in which labeled data in figure 1 : taxonomy of da as special case of transductive transfer learning (left). related problems (e.g., domain and out-of-distribution generalization) and da setups (1:1 and multi-source adaptation) (right). the target domain is absent, but unlabeled data might be abundant. uda thus provides an elegant and scalable solution. we believe these advances in uda will help for out-of-distribution generalization. a categorization of domain adaptation in nlp we categorize research into model-centric, datacentric and hybrid approaches, as shown in figure 1 . model-centric methods target approaches to augment the feature space, alter the loss function, the architecture or model parameters @xcite @xcite . data-centric methods focus on the data aspect and either involve pseudo-labeling (or bootstrapping) to bridge the domain gap @xcite @xcite , data selection @xcite @xcite and pre-training methods @xcite . as some approaches take elements of both, we include a hybrid category. 2 a comprehensive overview of uda methods and the tasks each method is applied to is provided in other surveys comprehensive reviews on da exist, each with a different focus: visual applications @xcite @xcite , machine translation (mt) @xcite , pre-neural da methods in nlp @xcite . seminal surveys in machine learning on transfer learning include pan and @xcite , @xcite , and @xcite . contributions in this survey, we (i) comprehensively review neural approaches to unsupervised domain adaptation in nlp, foot_2 (ii) we analyze and compare the strengths and weaknesses of the described approaches, and (iii) we outline potential challenges and future directions in this field. ## background first we introduce the classic learning paradigm with its core assumption, then we outline da setups. given {x 1 , ..., x n } = x the training instances and {y 1 , ..., y n } = y the corresponding class labels, the goal of machine learning is to learn a function f that generalizes well to unseen instances. in supervised learning, training data consists of tuples {(x i , y i )} n @xmath0, where n is the number of instances, while in unsupervised learning we only have {(x i )} n @xmath1. a general assumption in supervised machine learning is that the test data follows the same distribution as the training data. formally, training and test data are assumed to be independently and identically (i.i.d.) sampled from the same underlying distribution. in practice, this assumption does not hold, which translates into a drop in performance when the model f trained on a source domain s is tested on a different but related target domain t . ## model-centric approaches model-centric approaches redesign parts of the model: the feature space, the loss function or regularization and the structure of the model. we categorize them into feature-centric and loss-centric methods. ## data-centric methods recently, data-centric approaches are on a rise, due to rapid growth of data and the gain in popularity of pre-training methods. we summarize data-centric strands next, which differ whether they use pseudolabeling, select relevant data or use large unlabeled data or auxiliary tasks for model pre-training. ## hybrid approaches work on the intersection of data-centric and model-centric methods can be plentiful. it currently includes combining semi-supervised objectives with an adversarial loss @xcite , combining pivot-based approaches with pseudo-labeling @xcite and very recently with contextualized word embeddings @xcite , and combining multi-task approaches with domain shift @xcite , multi-task learning with pseudo-labeling (multi-task tritraining) @xcite , and adaptive ensembling @xcite , which uses a studentteacher network with a consistency-based self-ensembling loss and a temporal curriculum. they apply adaptive ensembling to study temporal and topic drift in political data classification @xcite . ## challenges and future directions while recent work has made important progress in neural uda, our survey reveals i) an overrepresentation and bias of work on sentiment analysis (cf. column bias in comprehensive uda benchmarks concretely, we recommend a) to create new benchmarks for uda with multiple tasks and of increasing complexity, setups beyond 1:1 adaptation, and datasets which document known variety facets of the data @xcite . this will help to learn about the known and unknown (section 3) as 'variety' (domain) matters; b) to release unlabeled data from the broader distribution from which annotated data was sampled, in line with @xcite ; this allows studying diachronic effects, as labeled evaluation data lacks diversity in terms of topics and time @xcite ; and c) to release unaggregated, multiple annotations to study divergences in annotations @xcite . back to the roots and how knowledge transfers revisiting classics in neural times is beneficial, as shown for example in recent work which brings back scl and pseudo-labeling methods (see x scarcity even unlabeled data can be scarce (x scarcity), particularly in highly-specialized language varieties (e.g., clinical data) @xcite . this is often due to data sharing restrictions. in some cases, only a trained source model could be available instead of raw or labeled texts @xcite . together with the quest for more efficient learning methods, the general question of how to adapt in light of x scarcity or absence becomes important. ## conclusion in this survey, we review strands of unsupervised domain adaptation, summarized into model-centric, data-centric, and hybrid methods, including trends in pre-training. we also revisit the notion of domain and suggest to use the term variety instead, to better capture the multitude of dimensions of variation. our survey identifies a limited focus on sentiment benchmarks and single-task evaluation for uda. lastly, we outline future directions, linking to the broader challenges related to learning beyond 1:1 scenarios and out-of-distribution generalization. this also calls for new directions on benchmarks and learning under scarce data.