PERL : Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models (2020)

Abstract: Pivot-based neural representation models have led to significant progress in domain adaptation for NLP. However, previous research following this approach utilize only labeled data from the source domain and unlabeled data from the source and target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. To alleviate this, we propose PERL: A representation learning model that extends contextualized word embedding models such as BERT (Devlin et al., 2019) with pivot-based fine-tuning. PERL outperforms strong baselines across 22 sentiment classification domain adaptation setups, improves in-domain model performance, yields effective reduced-size models, and increases model stability.1

## introduction natural language processing (nlp) algorithms are constantly improving, gradually approaching human-level performance @xcite @xcite . however, those algorithms often depend on the availability of large amounts of manually annotated data from the domain in which the task is performed. unfortunately, collecting such annotated data is often costly and laborious, which substantially limits the applicability of nlp technology. domain adaptation (da), training an algorithm on annotated data from a source domain so that it can be effectively applied to other target domains, is one of the ways to solve the above bottleneck. indeed, over the years substantial efforts have been devoted to the da challenge @xcite @xcite @xcite @xcite . our focus in this paper is on unsupervised da, the setup we consider most realistic. in this setup labeled data is available only from the source domain and unlabeled data is available from both the source and the target domains. while various approaches for da have been proposed ( §2), with the prominence of deep neural network (dnn) modeling, attention has been recently focused on representation learning approaches. within representation learning for unsupervised da, two approaches have been shown particularly useful. in one line of work, dnn-based methods that use compress-based noise reduction to learn cross-domain features have been developed @xcite . in another line of work, methods based on the distinction between pivot and nonpivot features @xcite learn a joint feature representation for the source and the target domains. later on, @xcite , and @xcite married the two approaches and achieved substantial improvements on a variety of da setups. despite their success, pivot-based dnn models still only utilize labeled data from the source domain and unlabeled data from both the source and the target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. with the recent game-changing success of contextualized word embedding models trained on such massive corpora @xcite , it is natural to ask whether information from such corpora can enhance these da methods, particularly that background knowledge from noncontextualized embeddings has shown useful for da @xcite . in this paper we hence propose an unsupervised da approach that extends leading approaches based on dnns and pivot-based ideas, so that they can incorporate information encoded in massive corpora ( §3). our model, named perl: pivotbased encoder representation of language, builds on massively pre-trained contextualized word embedding models such as bert @xcite . to adjust the representations learned by these models so that they close the gap between the source and target domains, we fine-tune their parameters using a pivot-based variant of the masked language modeling (mlm) objective, optimized on unlabeled data from both the source and the target domains. we further present r-perl (regularized perl), which facilitates parameter sharing for pivots with similar meaning. we perform extensive experimentation in various unsupervised da setups of the task of binary sentiment classification ( §4, 5). first, for compatibility with previous work, we experiment with the legacy product review domains of @xcite (12 setups). we then experiment with more challenging setups, adapting between the above domains and the airline review domain @xcite used in @xcite (4 setups), as well as the imdb movie review domain @xcite (6 setups). we compare perl to the best performing pivot-based methods @xcite and to da approaches that fine-tune a massively pretrained bert model by optimizing its standard mlm objective using target-domain unlabeled data @xcite . perl and r-perl substantially outperform these baselines, emphasizing the additive effect of massive pre-training and pivot-based fine-tuning. as an additional contribution, we show that pivot-based learning is effective beyond improving domain adaptation accuracy. particularly, we show that an in-domain variant of perl substantially improves the in-domain performance of a bert-based sentiment classifier, for varying training set sizes (from 100 to 20k labeled examples). we also show that perl facilitates the generation of effective reduced-size da models. finally, we perform an extensive ablation study ( §6) that uncovers perl's crucial design choices and demonstrates the stability of perl to hyper-parameter selection compared to other da methods. ## background and previous work there are several approaches to da, including instance re-weighting @xcite @xcite , subsampling from the participating domains @xcite and da through representation learning, where a joint representation is learned based on texts from the source and target domains @xcite @xcite . we first describe the unsupervised da pipeline, continue with representation learning methods for da with a focus on pivot-based methods, and, finally, describe contextualized embedding models. ## ablation analysis and discussion in order to shed more light on perl, we conduct an ablation analysis. we start by uncovering the hyper-parameters that have strong impact on its performance, and analyzing its stability across hyper-parameter configurations. we then explore the impact of some of the design choices we made when constructing the model. in order to keep our analysis concise and to avoid heavy computations, we have to consider only a handful of arbitrarily chosen da setups for each analysis. we follow the five-fold crossvalidation protocol of §4 for hyper-parameter tuning, except that in some of the analyses a hyper-parameter of interest is kept fixed. ## conclusions we presented perl, a domain-adaptation model that fine-tunes a massively pre-trained deep contextualized embedding encoder (bert) with a pivot-based mlm objective. perl outperforms strong baselines across 22 sentiment classification da setups, improves in-domain model performance, increases its cross-configuration stability and yields effective reduced-size models. our focus in this paper is on binary sentiment classification, as was done in a large body of previous da work. in future work we would like to extend perl's reach to structured (e.g., dependency parsing and aspect-based sentiment classification) and generation (e.g., abstractive summarization and machine translation) nlp tasks.