Unsupervised Domain Clusters in Pretrained Language Models (2020)

Abstract: The notion of “in-domain data” in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addition, domain labels are many times unavailable, making it challenging to build domain-specific systems. We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision – suggesting a simple data-driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and precision and recall with respect to an oracle selection.

## introduction it is common knowledge in modern nlp that using large amounts of high-quality training data is a key aspect in building successful machine-learning based systems. for this reason, a major challenge when building such systems is obtaining data in the domain of interest. but what defines a domain? natural language varies greatly across topics, styles, levels of formality, genres and many other linguistic nuances @xcite @xcite . this overwhelming diversity of language makes it hard to find the right data for the task, as it is nearly impossible to well-define the exact requirements from such data with respect to all the aforementioned aspects. on top of that, domain labels are usually unavailable -e.g. in large-scale web-crawled data like common crawl 1 which was recently used to it koran subtitles medical law bert-base-uncased train state-of-the-art pretrained language models for various tasks @xcite . domain data selection is the task of selecting the most appropriate data for a domain from a large corpus given a smaller set of in-domain data @xcite @xcite . in this work, we propose to use the recent, highly successful self-supervised pre-trained language models, e.g. @xcite ; @xcite for domain data selection. as pretrained lms demonstrate state-of-theart performance across many nlp tasks after being trained on massive amounts of data, we hypothesize that the robust representations they learn can be useful for mapping sentences to domains in an unsupervised, data-driven approach. we show that these models indeed learn to cluster sentence representations to domains without further supervision (e.g. figure 1 ), and quantify this phenomenon by fitting gaussian mixture models (gmms) to the learned representations and measuring the purity of the resulting unsupervised clustering. we then pro-pose methods to leverage these emergent domain clusters for domain data selection in two ways: our methods enable to select relevant data for the task while requiring only a small set of monolingual in-domain data. as they are based solely on the representations learned by self-supervised lms, they do not require additional domain labels which are usually vague and over-simplify the notion of domain in textual data. we evaluate our method on data selection for neural machine translation (nmt) using the multi-domain german-english parallel corpus composed by @xcite . our data selection methods enable to train nmt models that outperform those trained using the well-established cross-entropy difference method of @xcite across five diverse domains, achieving a recall of more than 95% in all cases with respect to an oracle that selects the "true" in-domain data. our contributions in this work are as follows. first, we show that pre-trained language models are highly capable of clustering textual data to domains with high accuracy in a purely unsupervised manner. second, we propose methods to select in-domain data based on this property using vectorspace retrieval and positive-unlabeled fine-tuning of pretrained language models for binary classification. third, we show the applicability of our proposed data selection methods on a popular benchmark for domain adaptation in machine translation. an additional contribution is a new, improved data split we create for this benchmark, as we point on issues with previous splits used in the literature. the code and data for this work is publicly available. 2 we hope this work will encourage more research on understanding the data landscape in nlp, enabling to "find the right data for the task" in the age of massive models and diverse data sources. ## neural machine translation in a multi-domain scenario as we showed that pre-trained language models are indeed very useful in clustering sentence representations by domains in an unsupervised manner, we now seek to harness this property for a down-stream task -domain data selection for machine translation. domain data selection is the task of selecting examples from a large corpus which are as close as possible to the domain of interest, given a smaller set of in-domain examples. the selected examples can be used to either (1) train a domainspecific model from scratch @xcite , (2) fine-tune a pre-trained general-domain model @xcite , or (3) prioritize data for annotation as in an active-learning framework, if only monolingual data is available @xcite . to demonstrate the need for domain data selection and set the stage for our data selection experiments, we perform preliminary experiments with nmt in a multi-domain scenario. ## domain data selection with pretrained language models as shown in the previous section, using the right data is critical for achieving good performance on an in-domain test set, and more data is not necessarily better. however, in real-world scenarios, the availability of data labeled by domain is limited, e.g. when working with large scale, web-crawled data. in this section we focus on a data-selection scenario where only a very small number of indomain sentences are used to select data from a larger unlabeled parallel corpus. an established method for data selection was proposed by @xcite , which was also used in training the winning systems in wmt 2019 @xcite . this method compares the cross-entropy, according to domain-specific and non-domain-specific language models, for each candidate sentence for selection. the sentences are then ranked by the cross-entropy difference, and only the top sentences are selected for training. while the method by moore and lewis (2010) is tried-and-true, it is based on simple n-gram language models which cannot generalize beyond the n-grams that are seen in the in-domain set. in addition, it is restricted to the in-domain and generaldomain datasets it is trained on, which are usually small. on the contrary, pre-trained language models are trained on massive amounts of text, and, as we showed through unsupervised clustering, learn representations with domain-relevant information. in the following sections, we investigate whether this property of pretrained language models makes them useful for domain data selection. ## related work previous works used n-gram lms for data selection @xcite or other count-based methods @xcite @xcite . while such methods work well in practice, they cannot generalize beyond the n-grams observed in the in-domain datasets, which are usually small. @xcite proposed to replace n-gram models with rnn-based lms with notable improvements. however, such methods do not capture the rich sentence-level global context as in the recent self-attention-based mlms; as we showed in the clustering experiments, autoregressive neural lms were inferior to masked lms in clustering the data by domain. in addition, training large lms may be prohibitive without relying on pre-training. regarding domain clustering for mt, hasler et al. (2014) discovered topics using lda instead of using domain labels. @xcite induced latent subdomains from the training data using a dedicated probabilistic model. many works used vector-based retrieval for data selection; @xcite learn to select data using bayesian optimization, and explored word2vec for that purpose. @xcite create paragraph vectors for data selection in the context of smt. @xcite use internal representations from the nmt model to perform data selection. @xcite propose a mechanism for incorporating retrieved sentences for each instance for domain adaptation in nmt, using representations extracted from a pretrained nmt model. @xcite explored instance-based data selection in a multi-domain scenario using information retrieval methods. other related works on domain adaptation include @xcite that adapts multi-domain nmt models with domain-aware feature embeddings, which are learned via an auxiliary language modeling task. @xcite proposed neuralnetwork based classifiers for data selection in smt. for more related work on data selection and domain adaptation in the context of mt, see the surveys by @xcite for smt and more recently @xcite for nmt. unrelated to mt, @xcite used bert to select data for tasks from the glue benchmark @xcite . however, they assumed supervision for all the different tasks/domains, while we propose an unsupervised method requiring only a small set of in-domain data. also in the context of pretrained language models, @xcite show the importance of additional pretraining with in-domain data to improve the downstream task-specific performance. while previous work made important contributions to domain data selection, our work is the first to explore massive pretrained language models for both unsupervised domain clustering and for data selection in nmt. ## conclusions and future work we showed that massive pre-trained language models are highly effective in mapping data to domains in a fully-unsupervised manner using averagepooled sentence representations and gmm-based clustering. we suggest that such clusters are a more appropriate, data driven approach to domains in natural language than simplistic labels (e.g. "medical text"), and that it will improve over time as better and larger pretrained lms will become available. we proposed new methods to harness this property for domain data selection using distance-based ranking in vector space and pretrained lm finetuning, requiring only a small set of in-domain data. we demonstrated the effectiveness of our methods on a new, improved data split we created for a previously studied multi-domain machine translation benchmark. our methods perform similarly or better than an established data selection method and oracle in-domain training across all five domains in the benchmark. this work just scratches the surface with what can be done on the subject; possible avenues for future work include extending this with multilingual data selection and multilingual lms @xcite @xcite , using such selection methods with domain-curriculum training @xcite , applying them on noisy, web-crawled data (junczys-dowmunt, 2018) or for additional tasks @xcite . another interesting avenue is applying this to unsupervised nmt, which is highly sensitive to domain mismatch @xcite . we hope this work will encourage more research on finding the right data for the task, towards more efficient and robust nlp. it koran subtitles medical law xlnet-base-cased it koran subtitles medical law gpt2 figure 7: 2d visualizations of the unsupervised gmm-based clustering for different pretrained auto-regressive lms.