Domain divergence plays a significant role in estimating the performance of a model when applied to new domains. While there is  significant literature on divergence measures, choosing an appropriate divergence measures remains difficult for researchers . We address this shortcoming by both surveying the literature and through an empirical study. We contribute a taxonomy of divergence measures consisting of three groups -- Information-theoretic, Geometric, and Higher-order measures -- and identify the relationships between them. We then ground the use of divergence measuresin three different application groups -- 1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild  . From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and higher-order measures are common for 2). To further help researchers , we validate these uses empirically through a correlation analysis of performance drops. We consider the current contextual word representations (CWR) to contrast with the older word distribution based representationsfor this analysis . We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective.
Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant literature on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application . We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes -- Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applications -- 1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild -- and use it URLanise our literature . From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To further help researchers choose appropriate measures to predict drop in performance -- an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word representations (CWR) and contrast with the older distributed representations . We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective.
R
when applied to
in
A
a
R
choosing
researchers find it hard to choose
R
measures remains difficult for researchers
for a given NLP application
R
contribute
develop
R
groups
classes
D
--
R
We then ground the use of divergence measuresin three different application groups
Further, to understand the common use-cases of these measures, we recognise three novel applications
A
-- and use it URLanise our literature
R
higher-order measures are
Higher-order measures are more
R
, we validate these uses empirically through a correlation analysis of performance drops. We
choose appropriate measures to predict drop in performance -- an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we
R
to
and
R
word distribution based representationsfor this analysis
distributed representations