Predicting the Success of Domain Adaptation in Text Similarity (2021)

Abstract: Transfer learning methods, and in particular domain adaptation, help exploit labeled data in one domain to improve the performance of a certain task in another domain. However, it is still not clear what factors affect the success of domain adaptation. This paper models adaptation success and selection of the most suitable source domains among several candidates in text similarity. We use descriptive domain information and cross-domain similarity metrics as predictive features. While mostly positive, the results also point to some domains where adaptation success was difficult to predict.

## introduction since the data-hungry deep learning models have beaten state-of-the-art performances in different natural language processing (nlp) tasks, many efforts have been made to deal with the scarcity of labeled data @xcite @xcite . one of the main avenues taken by researchers of this field is investigating the portability of models between different data distributions, often referred to as different domains @xcite . while multiple approaches have been proposed to make this portability feasible and efficient, it is still unclear how to predict the adaptability of two domains in advance. it is particularly important to address this gap because almost all domain adaptation approaches adjust a model to a new domain at the expense of more computational resources. therefore, in practice it is neither desirable nor scalable to try all possible dataset candidates. most relevant existing work seeks to identify key factors that can be used to justify why transfer learning between two domains work @xcite @xcite @xcite . in practice, however, one needs to be able to quantitatively select a set of existing datasets that can best be adapted to a certain domain for a certain task. we propose a simple yet effective approach to predict the success of transfer or adaptation, with the hope of drawing the research community's attention to this gap. we use the term domain transfer (dt) when a model trained on one domain is simply used for inference in another domain. domain adaptation (da) is used for approaches that bridge the source and target domain representations (e.g., by mapping or aligning feature spaces of the two domains) so that a model trained on labeled source data (and unlabeled target data) performs well in the target domain. while for the experiments in this paper we focus on the task of text similarity and autoencoder approaches to da, the proposed process, shown in figure 1 , can be easily applied to other nlp tasks and other unsupervised da approaches. ## data we use 11 publicly available semantic text similarity datasets. seven of them were obtained from stackexchange forums, with data from 2015 to november 2020: apple, askubuntu, math, stackoverflow, stats, superuser and unix foot_0 . we also use quora question pairs, microsoft research paraphrase corpus (mrpc) @xcite , paraphrase adversaries from word scrambling (paws) @xcite and sentences involving compositional knowledge-relatedness (sick) @xcite . all datasets contain binary labels indicating whether the text pair is similar or not. the exception is sick, where we convert the original relevance score of 1-5 into a binary score of 0 (not semantically similar) if the relevance score is below 4, and 1 otherwise. each of the 11 datasets is considered a separate domain. ## modeling domain transfer and adaptation figure 1 shows the process we use to implement dt and da and train a model that can best identify a proper source domain for a particular target domain. we start with embedding text in each of the 11 domains with the universal sentence encoder (use) @xcite . for dt, use representations are used directly to train models in a source domain s and evaluate the performance in the target domain t. for da, we implement both sda and msdar. a three-layer sda is trained on each source-target domain pair. text from the source and target domains is embedded with use and corrupted with gaussian noise, whose parameters are estimated from the hidden representation of the previous layer. in msdar, the hyperparameters we use are 5 layers, the target regularization parameter λ = 1, dropout probability 0.6 and the regularization objective @xmath0. these parameters have the same meaning as in @xcite . both sda and msdar encoders are then used to embed the use representations of the text in the source and target domains. figure 2 shows the original and msdar representations for stackoverflow and superuser domains, demonstrating the effect of msdar on aligning the feature spaces of the two domains. the representations described above are used to train a dense 3-layer neural network in the source domain and evaluate its performance in the target domain by reporting the f1-score. (we train three such models for each domain pair and average their performance.) we denote this cross-domain performance by f1 !" . in order to make the dt and da results robust to the relative difficulty of learning in different domains, we normalize f1 !" by the in-domain f1-score, f1 "" , which denotes the performance of the fully supervised model trained and evaluated in the same domain. the normalized f1-score, averaged over all domain pairs, is 0.775 for dt, 0.799 for sda and 0.817 for msdar. this is in line with previous work showing better performance of msdar over sda @xcite . ## domain similarity measures considering a source domain 𝑆 and a target domain 𝑇 with unigram sets 𝑈 ! and 𝑈 " , we define a set of features 𝐹 !" = {𝑓 # !" , . . . , 𝑓 #$ !" } as follows. unigram coverage. the simplest metric to evaluate the similarity of two domains is the percentage of their common unigrams. we use the ratio of common unigrams in the source ## discussion adaptation success factors. comparing cpr, top1, top3 and top5 between success predictor and domain ranker for all dt and da methods, we see that in general, domain ranker does a better job finding the orderings of candidate source domains. for success predictor, the features with highest importance are kl-divergence, 𝑓 / !" , target ratio, 𝑓 ) !" ,and data size of the target domain 𝑓 + !" . for domain ranker, average example lengths, ## conclusion and future work we studied the problem of selecting the most relevant labeled datasets from a pool of candidates to be used as a source domain in a transfer learning setup with a specific unlabeled target domain. the experiments focused on the text similarity task and autoencoder approaches to da. note that the proposed process can be extended to other nlp tasks and other unsupervised da approaches as well. we used descriptive domain information and cross-domain similarity metrics as predictive features to model the success of dt and da, and to rank source domains based on their relevancy. in future work, we intend to study source selection in multi-source domain adaptation setup, using multiple source domains for dt/da. identifying additional adaptation success factors that could better predict the success of dt/da for complex domains such as paws and quora, and learning the success threshold (here, we fixed it at 80%) are other avenues to investigate. other possibilities include experimenting with various text representations (such as bag-of-words) and models (e.g., transformer-based).