Domain Divergences: A Survey and Empirical Analysis (2021)

Abstract: Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant literature on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application. We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes — Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applications – 1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild – and use it to organise our literature. From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To further help researchers choose appropriate measures to predict drop in performance – an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word representations (CWR) and contrast with the older distributed representations. We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective.

## introduction standard machine learning models do not perform well when tested on data from a different target domain. the performance in a target domain largely depends on the domain divergence @xcite -a notion of distance between the two domains. thus, efficiently measuring and reducing divergence is crucial for adapting models to the new domain -the topic of domain adaptation. divergence also has practical applications in predicting the performance drop of a model when adapted to new domains @xcite , and in choosing among alternate models @xcite . given its importance, researchers have invested much effort to define and measure domain divergence. linguists use register variation to capture varieties in text -the difference between distributions of the prevalent features in two registers @xcite . other measures include probabilistic measures like h-divergence @xcite , information theoretic measures like jenssen-shannon and kullback-leibler divergence @xcite and measures using higher-order moments of random variables like maximum mean discrepancy (mmd) and central moment discrepancy (cmd) @xcite . the proliferation of divergence measures challenges researchers in choosing an appropriate measure for a given application. to help guide best practices, we first comprehensively review the nlp literature on domain divergences. unlike previous surveys, which focus on domain adaptation for specific tasks such as machine translation @xcite and statistical (non-neural network) models @xcite , our work takes a different perspective. we study domain adaptation through the vehicle of domain divergence measures. first, we develop a taxonomy of divergence measures consisting of three groups: information-theoretic, geometric, and higher-order measures. further, to find the most common group used in nlp, we recognise three novel application areas of these divergences -data selection, learning representations, and decisions in the wild and organise the literature under them. we find that information-theoretic measures over word distributions are popular for data selection and decisions in the wild, while higher-order measures over continuous features are frequent for learning representations. domain divergence is a major predictor of performance in the target domain. a better domain divergence metric ideally predicts the corresponding performance drop of a model when applied to a target domain -a practical and important component of decisions in the wild. we further help researchers identify appropriate measures for predicting performance drops, through a correlation analysis over 130 domain adaptation scenarios and three standard nlp tasks: part of speech tagging (pos), named entity recognition (ner), and sentiment analysis and 12 divergence metrics from our literature review. while information-theoretic measures over traditional word distributions are popular in the literature, are higher-order measures calculated over modern contextual word representations better indicators of performance drop? we indeed find that higher-order measures are superior, but traditional measures are still reliable indicators of performance drop. the closest to our work is @xcite who perform a correlation analysis. however, they do not compare against different divergence measures from the literature. comparatively, we consider more tasks and divergence measures. in summary, our contributions are: ## a taxonomy of divergence measures we devise a taxonomy for domain divergence measures, shown in figure 1 . it contains three main classes. individual measures belong to a single class, where relationships can exist between measures from different classes. we provide detailed description of individual measures in appendix a. geometric measures calculate the distance between two vectors in a metric space. as a divergence measure, they calculate the distance between features (tf.idf , continuous representations, etc.) extracted from instances of different domains. the p-norm is a generic form of the distance between two vectors, where manhattan (@xmath0. cosine (cos) uses the cosine of the angle between two vectors to measure similarity and 1-cos measures distance. geometric measures are easy to calculate, but are ineffective in a high dimensional space as all distances appear the same @xcite . information-theoretic measures captures the distance between probability distributions. for example, cross entropy over n-gram word distributions are extensively used in domain adaptation for machine translation. f -divergence @xcite is a general family of divergences where f is a convex function. different formulations of the f function lead to kl and js divergence. @xcite show that reducing f -divergence measure is equivalent to reducing the pad measures (see next section). another special case of f -divergence is the family of α divergences, where kl-div is a special case of α divergence. renyi divergence is a member of the α-divergences and tends towards kl-div as α → 1 (edge a ); often applied to optimal transport problems, wasserstein distance measures the amount of work needed to convert one probability distribution to the other as distance and is used extensively for domain adaptation. kl-div is also related to cross entropy (ce). in this paper, ce refers to measures based on entropy. higher-order measures consider matching higher order moments of random variables or divergence in a projected space. their properties are amenable to end-to-end learning based domain adaptation and recently have been extensively adopted. maximum mean discrepancy (mmd) is one such measure which considers matching first order moments of variables in a reproducible kernel hilbert space. on the other hand, coral @xcite considers second order moments and cmd @xcite considers higher order moments. coral and cmd are desirable because they avoid computationally expensive kernel matrix computations. kl-div can also be considered as matching the first-order moment @xcite ; edge b . proxy-a-distance (pad) measures the distance between source and target distributions via the error of a classifier in target domain samples as source domain samples @xcite . a few other measures do not have ample support in the literature. these include informationtheoretic measures such as bhattacharya coefficient, higher-order measures like pad* @xcite , word vector variance (wvv), and term vocabulary overlap (tvo) @xcite . our taxonomy synthesises the diversity and figure 1 : taxonomy for divergence measures. i) geometric measures the distance between vectors in a metric space ii) information-theoretic measures the distance between probability distributions and iii) higher-order measures the distance between distributions considering higher moments or the distance between representations or their projections in a nonlinear space. edge a indicates that renyi divergence tends towards kl divergence as α → 1 and edge b indicates kl-div can be considered as matching first-order moment. ## applications of divergence measures our key observation of the literature is that there are three primary families of applications of divergences (cf. we limit the scope our survey to works that focus on divergence measures. we only consider unsupervised domain adaptation (uda) -where there is no annotated data available in the target domain. it is more practical yet more challenging. for a complete treatment of neural networks and uda in @xcite . also, we do not treat multilingual work. while cross-lingual transfer can be regarded as an extreme form of domain adaptation, measuring the distance between languages requires different divergence measures, outside our purview. ## experiments a practical use case of domain divergences is to predict the performance drop of a model applied to a new domain. we ask how relevant are traditional measures over word distributions compared to higher-order measures like cmd and mmd over contextual word representations like bert, elmo, distilbert @xcite @xcite ? we perform an empirical study to assess their suitability to predict performance drops for three important nlp tasks: pos, ner, and sa leaving machine translation to future work. performance difference between the source and the target domain depends on the divergence between their feature distributions @xcite . we assume a co-variate shift, as in @xcite , where the marginal distribution over features change, but the conditional label distributions does not -i.e., p ds (y|x) = p d t (y|x) p ds (x) = p d t (x). although difference in conditional label distribution can increase the h-divergence measure @xcite , it requires labels in the target domain for assessment. in this work, we assume no labelled data in the target domain, to best mimic realistic settings. ## conclusion we survey domain adaptation works, focusing on divergence measures and their usage for data selection, learning domain-invariant representations, and making decisions in the wild. we synthesised the divergence measures into a taxonomy of information theoretic, geometric and higher-order measures. while traditional measures are common for data selection and making decisions in the wild, higher-order measures are prevalent in learning representations. based on our correlation experiments, silhouette scores, and t-sne plots, we make the following recommendations: it is best used when there are sufficient examples to train a domain discriminator.