Original: A CTIVE L EARNING OVER M ULTIPLE D OMAINS IN N ATURAL L ANGUAGE T ASKS
Revised: A CTIVE L EARNING OVER M ULTIPLE D OMAINS IN N ATURAL L ANGUAGE T ASKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Studies of active learning traditionally assume the target and source data stem from a single domain.
Revised: Studies of active learning traditionally assume the target and source data stem from a single domain.

Original: However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear a priori which data sources will help or hurt the target domain.
Revised: However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear a priori which data sources will help or hurt the target domain.

Original: We survey a wide variety of techniques in active learning (AL), domain shift detection (DS) and multi-domain sampling to examine this challenging setting for question answering and sentiment analysis.
Revised: We survey a wide variety of techniques in active learning (AL), domain shift detection (DS) and multi-domain sampling to examine this challenging setting for question answering and sentiment analysis.

Original: We ask (1) what family of methods are effective for this task?
Revised: We ask (1) what family of methods are effective for this task?

Original: And, (2) what properties of selected examples and domains achieve strong results?
Revised: And, (2) what properties of selected examples and domains achieve strong results?

Original: Among 18 acquisition functions from 4 families of methods, we ﬁnd HDivergence methods, and particularly our proposed variant DAL-E, yield effective results, averaging 2-3% improvements over the random baseline.
Revised: Among 18 acquisition functions from 4 families of methods, we ﬁnd HDivergence methods, and particularly our proposed variant DAL-E, yield effective results, averaging 2-3% improvements over the random baseline.

Original: We also show the importance of a diverse allocation of domains, as well as room-for-improvement of existing methods on both domain and example selection.
Revised: We also show the importance of a diverse allocation of domains, as well as room-for-improvement of existing methods on both domain and example selection.

Original: Our ﬁndings yield the ﬁrst comprehensive analysis of both existing and novel methods for practitioners faced with multi-domain active learning for natural language tasks.
Revised: Our ﬁndings yield the ﬁrst comprehensive analysis of both existing and novel methods for practitioners faced with multi-domain active learning for natural language tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: New natural language problems, outside the watershed of core NLP, are often strictly limited by a dearth of labeled data.
Revised: New natural language problems, outside the watershed of core NLP, are often strictly limited by a dearth of labeled data.

Original: While unlabeled data is frequently available, it is not always from the same source as the target distribution.
Revised: While unlabeled data is frequently available, it is not always from the same source as the target distribution.

Original: This is particularly prevalent for tasks characterized by (i) signiﬁcant distribution shift over time, (ii) personalization for user subgroups, or (iii) different collection mediums (e.g. changes to annotator instructions, or spoken vs. typed user inputs).
Revised: This is particularly prevalent for tasks characterized by (i) signiﬁcant distribution shift over time, (ii) personalization for user subgroups, or (iii) different collection mediums (see examples in Section A).

Original: A widely-used solution to this problem is to bootstrap a larger training set using active learning (AL): a method to decide which unlabeled training examples should be labeled on a ﬁxed annotation budget (Cohn et al., 1996; Settles, 2012).
Revised: A widely-used solution to this problem is to bootstrap a larger training set using active learning (AL): a method to decide which unlabeled training examples should be labeled on a ﬁxed annotation budget (Cohn et al., 1996; Settles, 2012).

Original: However, most active learning literature in NLP assumes the unlabeled source data is drawn from the same distribution as the target data (Dor et al., 2020).
Revised: However, most active learning literature in NLP assumes the unlabeled source data is drawn from the same distribution as the target data (Dor et al., 2020).

Original: This simplifying assumption avoids the frequent challenges faced by practitioners in multi-domain active learning .
Revised: This simplifying assumption avoids the frequent challenges faced by practitioners in multi-domain active learning .

Original: In this realistic setting, there are multiple sources of data ( i.e. domains) to consider.
Revised: In this realistic setting, there are multiple sources of data ( i.e. domains) to consider.

Original: In this case, it’s unclear whether to optimize for homogeneity or heterogeneity of selected examples.
Revised: In this case, it’s unclear whether to optimize for homogeneity or heterogeneity of selected examples.

Original: Secondly, is it more effective to allocate an example budget per domain, or treat examples as a single unlabeled pool?
Revised: Secondly, is it more effective to allocate an example budget per domain, or treat examples as a single unlabeled pool?

Original: Where active learning baselines traditionally select examples the model is least conﬁdent on (Settles, 2009), in this setting it could lead to distracting examples from very dissimilar distributions.
Revised: Where active learning baselines traditionally select examples the model is least conﬁdent on (Settles, 2009), in this setting it could lead to distracting examples from very dissimilar distributions.

Original: In this work we empirically examine four separate families of methods (uncertainty-based, HDivergence, reverse classiﬁcation accuracy, and semantic similarity detection) over several question answering and sentiment analysis datasets, following (Lowell et al., 2019; Elsahar & Gall´e, 2019b), to provide actionable insights to practitioners facing this challenging variant of active learning for natural language.
Revised: In this work we empirically examine four separate families of methods (uncertainty-based, HDivergence, reverse classiﬁcation accuracy, and semantic similarity detection) over several question answering and sentiment analysis datasets, following (Lowell et al., 2019; Elsahar & Gall´e, 2019b), to provide actionable insights to practitioners facing this challenging variant of active learning for natural language.

Original: We address the following questions:
Revised: We address the following questions:

Original: What family of methods are effective for multi-domain active learning?
Revised: What family of methods are effective for multi-domain active learning?

Original: What properties of the example and domain selection yield strong results?
Revised: What properties of the example and domain selection yield strong results?

Original: While previous work has investigated similar settings (Saha et al., 2011; Liu et al., 2015; Zhao et al., 2021;
Revised: While previous work has investigated similar settings (Saha et al., 2011; Liu et al., 2015; Zhao et al., 2021;

Original: Kirsch et al., 2021) we contribute, to our knowledge, the ﬁrst rigorous formalization and broad survey of methods within NLP.
Revised: Kirsch et al., 2021) we contribute, to our knowledge, the ﬁrst rigorous formalization and broad survey of methods within NLP.

Original: We ﬁnd that many families of techniques for active learning and domain shift detection fail to reliably beat random baselines in this challenging variant of active learning, but certain H -Divergence methods are consistently strong.
Revised: We ﬁnd that many families of techniques for active learning and domain shift detection fail to reliably beat random baselines in this challenging variant of active learning, but certain H -Divergence methods are consistently strong.

Original: Our analysis identiﬁes stark dissimilarities of these methods’ example selection, and suggests domain diversity is an important factor in achieving strong results.
Revised: Our analysis identiﬁes stark dissimilarities of these methods’ example selection, and suggests domain diversity is an important factor in achieving strong results.

Original: These results may serve as a guide to practitioners facing this problem, suggesting particular methods that are generally effective and properties of strategies that increase performance.
Revised: These results may serve as a guide to practitioners facing this problem, suggesting particular methods that are generally effective and properties of strategies that increase performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Active Learning in NLP Lowell et al.
Revised: Active Learning in NLP Lowell et al.

Original: (2019) shows how inconsistent active learning methods are in NLP, even under regular conditions.
Revised: (2019) shows how inconsistent active learning methods are in NLP, even under regular conditions.

Original: However, Dor et al.
Revised: However, Dor et al.

Original: (2020); Siddhant & Lipton (2018) survey active learning methods in NLP and ﬁnd notable gains over random baselines.
Revised: (2020); Siddhant & Lipton (2018) survey active learning methods in NLP and ﬁnd notable gains over random baselines.

Original: Kouw & Loog (2019) survey domain adaptation without target labels, similar to our setting, but for non-language tasks.
Revised: Kouw & Loog (2019) survey domain adaptation without target labels, similar to our setting, but for non-language tasks.

Original: We reference more active learning techniques in Section 4.
Revised: We reference more active learning techniques in Section 4.

Original:  Shift Detection Elsahar & Gall´e (2019b) attempt to predict accuracy drops due to domain shifts and Rabanser et al.
Revised: Domain Shift Detection Elsahar & Gall´e (2019b) attempt to predict accuracy drops due to domain shifts and Rabanser et al.

Original: (2018) surveys different domain shift detection methods.
Revised: (2018) surveys different domain shift detection methods.

Original: Arora et al.
Revised: Arora et al.

Original: (2021) examine calibration and density estimation for textual OOD detection.
Revised: (2021) examine calibration and density estimation for textual OOD detection.

Original: Active Learning under Distribution Shift
Revised: Active Learning under Distribution Shift

Original: A few previous works investigated active learning under distribution shifts, though mainly in image classiﬁcation, with single source and target domains.
Revised: A few previous works investigated active learning under distribution shifts, though mainly in image classiﬁcation, with single source and target domains.

Original: Kirsch et al.
Revised: Kirsch et al.

Original: (2021) ﬁnds that BALD, which is often considered the state of the art for unshifted domain settings, can get stuck on irrelevant source domain or junk data.
Revised: (2021) ﬁnds that BALD, which is often considered the state of the art for unshifted domain settings, can get stuck on irrelevant source domain or junk data.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2021) investigates label shift , proposing a combination of predicted class balanced subsampling and importance weighting.
Revised: (2021) investigates label shift , proposing a combination of predicted class balanced subsampling and importance weighting.

Original: Saha et al.
Revised: Saha et al.

Original: (2011), whose approach corrects joint distribution shift, relies on the covariate shift assumption .
Revised: (2011), whose approach corrects joint distribution shift, relies on the covariate shift assumption .

Original: However, in practical settings, there may be general distributional shifts where neither the covariate shift nor label shift assumptions hold.
Revised: However, in practical settings, there may be general distributional shifts where neither the covariate shift nor label shift assumptions hold.

Original: Transfer Learning from Multiple Domains Attempts to better understand how to handle shifted domains for better generalization or target performance has motivated work in question answering (Talmor & Berant, 2019;
Revised: Transfer Learning from Multiple Domains Attempts to better understand how to handle shifted domains for better generalization or target performance has motivated work in question answering (Talmor & Berant, 2019;

Original: Fisch et al., 2019; Longpre et al., 2019; Kamath et al., 2020) and classiﬁcation tasks (Ruder & Plank, 2018; Sheoran et al., 2020).
Revised: Fisch et al., 2019; Longpre et al., 2019; Kamath et al., 2020) and classiﬁcation tasks (Ruder & Plank, 2018; Sheoran et al., 2020).

Original: Ruder & Plank (2017) show the beneﬁts of both data similarity and diversity in transfer learning.
Revised: Ruder & Plank (2017) show the beneﬁts of both data similarity and diversity in transfer learning.

Original: R ¨ uckl ´ e et al.
Revised: R¨uckl´e et al.

Original: ﬁnd that sampling from a wide-variety of source domains (data scale) outperforms sampling similar domains in question answering.
Revised: ﬁnd that sampling from a widevariety of source domains (data scale) outperforms sampling similar domains in question answering.

Revised: He et al.

Revised: (2021) investigate a version of multi-domain active learning where models are trained and evaluated on examples from all domains, focusing on robustness across domains.

Original:  M ULTI -D OMAIN A CTIVE L EARNING
Revised:  M ULTI -D OMAIN A CTIVE L EARNING

Original: Suppose we have multiple domains D 1 , D 2 , ..., D k .
Revised: Suppose we have multiple domains D 1 , D 2 , ..., D k .

Original: Let one of the k domains be the target set D T , and let the other k − 1 domains comprise the source set D S
Revised: Let one of the k domains be the target set D T , and let the other k − 1 domains comprise the source set D S

Original: = (cid:83) i (cid:54) =
Revised: = (cid:83) i (cid:54) =

Original: T D i .
Revised: T D i .

Original: Given:
Revised: Given:

Original: • Target:
Revised: • Target:

Original: Small samples of labeled data points ( x, y ) from the target domain.
Revised: Small samples of labeled data points ( x, y ) from the target domain.

Original: [Equation]
Revised: D trainT , D devT , D testT ∼ D T .•

Original: • Source: A large sample of unlabeled points ( x ) from the source domains.
Revised: Source: A large sample of unlabeled points ( x ) from the source domains.

Original: [Equation]
Revised: D S = (cid:83) i (cid:54) = T D i

Original: Task:
Revised: Task:

Original: Choose n samples from D S to label.
Revised: Choose n samples from D S to label.

Original: D chosenS ⊂ D S , | D chosenS | = n , selected by arg max x ∈ D S
Revised: D chosenS ⊂ D S , | D chosenS | = n , selected by arg max x ∈ D S

Original: A f ( x ) where A f is an acquisition function: a policy to select unlabeled examples from D S for labeling.
Revised: A f ( x ) where A f is an acquisition function: a policy to select unlabeled examples from D S for labeling.

Original: Train a model M on D final − train , validating on D dev T .
Revised: Train a model M on D final − train , validating on D dev T .

Original: D final − train =
Revised: D final − train =

Original: D trainT ∪ D chosenS 3.
Revised: D trainT ∪ D chosenS 3.

Original: Evaluate M on D testT , giving score s .
Revised: Evaluate M on D testT , giving score s .

Original: For Step 1, the practitioner chooses n samples with the highest scores according to their acquisition function A f .
Revised: For Step 1, the practitioner chooses n samples with the highest scores according to their acquisition function A f .

Original: M is ﬁne-tuned on these n samples, then evaluated on D testT to demonstrate A f ’s ability to choose relevant out-of-distribution training examples.
Revised: M is ﬁne-tuned on these n samples, then evaluated on D testT to demonstrate A f ’s ability to choose relevant out-of-distribution training examples.

Original:  M ETHODS
Revised:  M ETHODS

Original: We identify four families of methods relevant to active learning over multiple shifted domains.
Revised: We identify four families of methods relevant to active learning over multiple shifted domains.

Original: Uncertainty methods are common in standard active learning for measuring example uncertainty or familiarity to a model; H -Divergence techniques train classiﬁers for domain shift detection;
Revised: Uncertainty methods are common in standard active learning for measuring example uncertainty or familiarity to a model; H -Divergence techniques train classiﬁers for domain shift detection;

Original: Semantic Similarity Detection ﬁnds data points similar to points in the target domain; and Reverse
Revised: Semantic Similarity Detection ﬁnds data points similar to points in the target domain; and Reverse

Original: Classiﬁcation Accuracy approximates the beneﬁt of training on a dataset.
Revised: Classiﬁcation Accuracy approximates the beneﬁt of training on a dataset.

Revised: A limitation of our work is we do not cover all method families, such as domain adaptation, just those we consider most applicable.

Original: We derive ∼ 18 active learning variants, comprising the most prevalent and effective from prior work, and novel extensions/variants of existing paradigms for the multi-domain active learning setting (see K NN, (cid:93) RCA and DAL-E).
Revised: We derive ∼ 18 active learning variants, comprising the most prevalent and effective from prior work, and novel extensions/variants of existing paradigms for the multi-domain active learning setting (see K NN, (cid:93) RCA and DAL-E).

Original: Furthermore, we split the families into two acquisition strategies: Single Pool Strategy and Domain
Revised: Furthermore, we split the families into two acquisition strategies: Single Pool Strategy and Domain

Original: Budget Allocation .
Revised: Budget Allocation .

Original: Single Pool Strategy , comprising the ﬁrst three families of methods, treats all examples as coming from one single unlabeled pool.
Revised: Single Pool Strategy , comprising the ﬁrst three families of methods, treats all examples as coming from one single unlabeled pool.

Original: Domain Budget Allocation , consisting of
Revised: Domain Budget Allocation , consisting of

Original: Reverse Classiﬁcation Accuracy methods, simply allocate an example budget for each domain.
Revised: Reverse Classiﬁcation Accuracy methods, simply allocate an example budget for each domain.

Original: We enumerate acquisition methods A f below.
Revised: We enumerate acquisition methods A f below.

Original: Each method produces a full ranking of examples in the source set D S .
Revised: Each method produces a full ranking of examples in the source set D S .

Original: To rank examples, most acquisition methods train an acquisition model, M A , using the same model architecture as M .
Revised: To rank examples, most acquisition methods train an acquisition model, M A , using the same model architecture as M .

Original: M A is trained on all samples from D trainT , except for DAL and K NN, which split D trainT into two equal segments, one for training M A and one for an internal model.
Revised: M A is trained on all samples from D trainT , except for DAL and K NN, which split D trainT into two equal segments, one for training M A and one for an internal model.

Original: Some methods have both ascending and descending orders of these rankings (denoted by ↑ and ↓ respectively, in the method abbreviations), to test whether similar or distant examples are preferred in a multi-domain setting.
Revised: Some methods have both ascending and descending orders of these rankings (denoted by ↑ and ↓ respectively, in the method abbreviations), to test whether similar or distant examples are preferred in a multi-domain setting.

Original: Certain methods use vector representations of candidate examples.
Revised: Certain methods use vector representations of candidate examples.

Original: We benchmark with both taskagnostic and task-speciﬁc encoders.
Revised: We benchmark with both taskagnostic and task-speciﬁc encoders.

Original: The task-agnostic embeddings are taken from the last layer’s
Revised: The task-agnostic embeddings are taken from the last layer’s

Original: CLS token in Reimers & Gurevych (2019)’s sentence encoder (Appendix for details).
Revised: CLS token in Reimers & Gurevych (2019)’s sentence encoder (Appendix for details).

Original: The task-speciﬁc embeddings are taken from the last layer’s
Revised: The task-speciﬁc embeddings are taken from the last layer’s

Original: CLS token in the trained model M A .
Revised: CLS token in the trained model M A .

Original: The motivation of the task-speciﬁc variant is that each example’s representation will capture taskrelevant differences between examples while ignoring irrelevant differences.
Revised: The motivation of the task-speciﬁc variant is that each example’s representation will capture taskrelevant differences between examples while ignoring irrelevant differences.

Original: The versions of DAL and K NN methods that use task-speciﬁc vectors are denoted with “ ∗ ” in their abbreviation.
Revised: The versions of DAL and K NN methods that use task-speciﬁc vectors are denoted with “ ∗ ” in their abbreviation.

Original: Otherwise, they use task-agnostic vectors.
Revised: Otherwise, they use task-agnostic vectors.

Original: 4. U NCERTAINTY M ETHODS
Revised: 4. U NCERTAINTY M ETHODS

Original: These methods measure the uncertainty of a trained model on a new example.
Revised: These methods measure the uncertainty of a trained model on a new example.

Original: Uncertainty can reﬂect either aleatoric uncertainty, due to ambiguity inherent in the example, or epistemic uncertainty, due to limitations of the model (Kendall & Gal, 2017).
Revised: Uncertainty can reﬂect either aleatoric uncertainty, due to ambiguity inherent in the example, or epistemic uncertainty, due to limitations of the model (Kendall & Gal, 2017).

Original: For the following methods, let Y be the set of all possible labels produced from the model M ( x ) and l y be the logit value for y ∈ Y .
Revised: For the following methods, let Y be the set of all possible labels produced from the model M ( x ) and l y be the logit value for y ∈ Y .

Original: Conﬁdence (CONF)
Revised: Conﬁdence (CONF)

Original: A model’s conﬁdence P ( y | x ) in its prediction y estimates the difﬁculty or unfamiliarity of an example (Guo et al., 2017; Elsahar & Gall´e, 2019a).
Revised: A model’s conﬁdence P ( y | x ) in its prediction y estimates the difﬁculty or unfamiliarity of an example (Guo et al., 2017; Elsahar & Gall´e, 2019a).

Original: Entropy (ENTR) Entropy applies Shannon entropy (Shannon, 1948) to the full distribution of class probabilities for each example, formalized as A ENTR .
Revised: Entropy (ENTR) Entropy applies Shannon entropy (Shannon, 1948) to the full distribution of class probabilities for each example, formalized as A ENTR .

Original: [Equation]
Revised: [Equation]

Original: Energy-based Out-of-Distribution Detection (ENG)
Revised: Energy-based Out-of-Distribution Detection (ENG)

Original: Liu et al.
Revised: Liu et al.

Original: (2020) use an energy-based score to distinguish between in- and out-distribution examples.
Revised: (2020) use an energy-based score to distinguish between in- and out-distribution examples.

Original: They demonstrate this method is less susceptible to overconﬁdence issues of softmax approaches.
Revised: They demonstrate this method is less susceptible to overconﬁdence issues of softmax approaches.

Original: Bayesian Active Learning by Disagreement (BALD) Houlsby et al. (2011) introduce Bayesian Active Learning byDisagreement, an alternative estimate of uncertainty, measuring prediction disagreement over multiple inference passes, each with a distinct dropout mask.
Revised: Bayesian Active Learning by Disagreement (BALD) Gal & Ghahramani (2016) introduces estimating uncertainty by measuring prediction disagreement over multiple inference passes, each with a distinct dropout mask.

Original: BALD isolates epistemic uncertainty, as the model would theoretically produce stable predictions over inference passes given sufﬁcient capacity.
Revised: BALD isolates epistemic uncertainty, as the model would theoretically produce stable predictions over inference passes given sufﬁcient capacity.

Original: We conduct T = 20 forward passes on x .
Revised: We conduct T =forward passes on x .

Original: ˆ y t
Revised: ˆ y t

Original: = argmax i
Revised: = argmax i

Original: P
Revised: P

Original: ( y i | x ) t , representing the predicted class on the t -th model pass on x .
Revised: ( y i | x ) t , representing the predicted class on the t -th model pass on x .

Original: Following (Lowell et al., 2019), ties are broken by taking the mean label entropy over all T runs.
Revised: Following (Lowell et al., 2019), ties are broken by taking the mean label entropy over all T runs.

Original: [Equation]
Revised: [Equation]

Original: 4. H -D IVERGENCE M ETHODS
Revised: 4. H -D IVERGENCE M ETHODS

Original: Ben-David et al.
Revised: Ben-David et al.

Original: (2006; 2010) formalize the divergence between two domains as the H -Divergence, which they approximate as the difﬁculty for a discriminator to differentiate between the two.
Revised: (2006; 2010) formalize the divergence between two domains as the H -Divergence, which they approximate as the difﬁculty for a discriminator to differentiate between the two.

Original: Discriminative Active Learning (DAL) applies this concept to the active learning setting (Gissin &
Revised: Discriminative Active Learning (DAL) applies this concept to the active learning setting (Gissin &

Original: Shalev-Shwartz, 2019).
Revised: Shalev-Shwartz, 2019).

Original: We explore variants of DAL, using an XGBoost decision tree (Chen & Guestrin, 2016) as the discriminator model g . 

Revised: We explore variants of DAL, using an XGBoost decision tree (Chen & Guestrin, 2016) as the discriminator model g .

Original: For the following methods, let D train − B T be the 1k examples from D trainT that were not used to train M A .
Revised: For the following methods, let D train − B T be the 1k examples from D trainT that were not used to train M A .

Original: Let E be an encoder function, which can be task-speciﬁc or agnostic as described above.
Revised: Let E be an encoder function, which can be task-speciﬁc or agnostic as described above.

Original: We use samples both from D train − B T and D S to train the discriminator.
Revised: We use samples both from D train − B T and D S to train the discriminator.

Original: We assign samples origin labels l , which depend on the DAL variant.
Revised: We assign samples origin labels l , which depend on the DAL variant.

Original: Samples from D S with discriminator predictions closest to 1 are selected for labeling.
Revised: Samples from D S with discriminator predictions closest to 1 are selected for labeling.

Original: The acquisition scoring function for each DAL method and training set deﬁnition, respectively, are:
Revised: The acquisition scoring function for each DAL method and training set deﬁnition, respectively, are:

Original: [Equation]
Revised: [Equation]

Revised: Discriminative Active

Original: Discriminative Active Learning — Target (DAL-T) DAL-T trains a discriminator g to distinguish between target examples in D train − B T and out-of-distribution examples from D 
Revised: Learning — Target (DAL -T) DAL-T trains a discriminator g to distinguish between target examples in D train − B T and out-of-distribution examples from D S .

Original: S .

Original: For DAL-T, l = 1 D train − B T ( x ) .
Revised: For DAL-T, l = 1 D train − B T ( x ) .

Original: Discriminative Active Learning — Error (DAL-E)
Revised: Discriminative Active Learning — Error (DAL-E)

Original: DAL-E is a novel variant of DAL.
Revised: DAL-E is a novel variant of DAL.

Original: DALE’s approach is to ﬁnd examples that are similar to those in the target domain that M A misclassiﬁed.
Revised: DALE’s approach is to ﬁnd examples that are similar to those in the target domain that M A misclassiﬁed.

Original: We partition D train − B T further into erroneous samples
Revised: We partition D train − B T further into erroneous samples

Original: D
Revised: D

Original: errT and correct samples D corrT , where
Revised: errT and correct samples D corrT , where

Original: D train − B T =
Revised: D train − B T =

Original: D errT ∪ D corrT .
Revised: D errT ∪ D corrT .

Original: For DAL-E, l = 1 D errT ( x ) .
Revised: For DAL-E, l = 1 D errT ( x ) .

Original: 4. R EVERSE C LASSIFICATION A CCURACY 
Revised: 4. R EVERSE C LASSIFICATION A CCURACY RCA

Original: RCA

Original: Reverse Classiﬁcation Accuracy (RCA) estimates how effective source set D i,i ∈ S is as a training data for target test set D T (Fan & Davidson, 2006; Elsahar & Gall ´ e, 2019b).
Revised: Reverse Classiﬁcation Accuracy (RCA) estimates how effective source set D i,i ∈ S is as a training data for target test set D T (Fan & Davidson, 2006; Elsahar & Gall´e, 2019b).

Original: Without gold labels for D
Revised: Without gold labels for D

Original: i we compute soft labels instead, using the BERT-Base M A trained on the small labeled set D trainT .
Revised: i we compute soft labels instead, using the BERT-Base M A trained on the small labeled set D trainT .

Original: We then train a child model M i on D i using these soft labels, and evaluate the child model on D devT .
Revised: We then train a child model M i on D i using these soft labels, and evaluate the child model on D devT .

Original: RCA chooses examples randomly from whichever domain i produced the highest score
Revised: RCA chooses examples randomly from whichever domain i produced the highest score

Original: s i .
Revised: s i .

Original: [Equation]
Revised: [Equation]

Original: RCA-Smoothed ( (cid:93) RCA )
Revised: RCA-Smoothed ( (cid:93) RCA )

Original: Standard RCA only selects examples from one domain D i .
Revised: Standard RCA only selects examples from one domain D i .

Original: We develop a novel variant which samples from multiple domains, proportional to their relative performance on the target domain D devT .
Revised: We develop a novel variant which samples from multiple domains, proportional to their relative performance on the target domain D devT .

Original: RCA-smoothed ( (cid:93) RCA ) selects | D choseni | examples from source domain i , based on the relative difference between the performance
Revised: RCA-smoothed ( (cid:93) RCA ) selects | D choseni | examples from source domain i , based on the relative difference between the performance

Original: s i (of child model M i trained on domain i with pseudo-labels from M A ) on the target domain, and the performance s T of a model trained directly on the target domain D devT .
Revised: s i (of child model M i trained on domain i with pseudo-labels from M A ) on the target domain, and the performance s T of a model trained directly on the target domain D devT .

Original: Since these strategies directly estimates model performance on the target domain resulting from training on each source domain, RCA and (cid:93) RCA are strong Domain
Revised: Since these strategies directly estimates model performance on the target domain resulting from training on each source domain, RCA and (cid:93) RCA are strong Domain

Original: Budget Allocation candidates.
Revised: Budget Allocation candidates.

Original: 4. N EAREST N EIGHBOUR / S EMANTIC S IMILARITY D ETECTION ( K NN

Original: Nearest neighbour methods ( K NN) are used to ﬁnd examples that are semantically similar.

Original: Using sentence encoders we can search the source set D S to select the top k nearest examples by cosine

Original: [Table]
Revised: [Table]

Original: Sentiment Datasets
Revised: Sentiment Datasets

Original: Dataset
Revised: Dataset

Original: Q
Revised: Q

Original: [Table]
Revised: [Table]

Original: |
Revised: |

Original: [Table]
Revised: [Table]

Original: |
Revised: |

Revised: Q

Original: [Table]
Revised: [Table]

Original: Dataset | R | -
Revised: Dataset | R | -

Original: N +
Revised: N +

Original: SQuAD
Revised: SQuAD

Original: Crowd
Revised: Crowd

Original: Wiki [Table]
Revised: Wiki (cid:55)

Original: Amzn
Revised: Amzn

Original: [Table]
Revised: [Table]

Original: Books 12.
Revised: Books 12.

Original: [Table]
Revised: NewsQA

Original: Crowd
Revised: Crowd

Original: News (cid:51)
Revised: News (cid:51)

Revised: Amzn

Original: [Table]
Revised: [Table]

Original: Health 9.
Revised: Health 9.

Original: TriviaQA
Revised: TriviaQA

Original: Trivia
Revised: Trivia

Original: Web (cid:51)
Revised: Web (cid:51)

Original: Amzn [Table] Music 36.
Revised: Amzn-Music 36.

Original: SearchQA
Revised: SearchQA

Original: Jeopardy
Revised: Jeopardy

Original: Web [Table]
Revised: Web (cid:51)

Original: Amzn
Revised: Amzn

Original: [Table]
Revised: [Table]

Original: Software 14.
Revised: Software 14.

Original: HotpotQA
Revised: HotpotQA

Original: Crowd
Revised: Crowd

Original: Wiki [Table]
Revised: Wiki (cid:55)

Original: Amzn-Sports 49.
Revised: Amzn-Sports 49.

Original: Natural
Revised: Natural-

Original: [Table]
Revised: [Table]

Original: Search
Revised: Search

Original: Wiki (cid:51)
Revised: Wiki (cid:51)

Original: Amzn-Tools 15.
Revised: Amzn-Tools 15.

Original: Imdb 16.
Revised: Imdb 16.

Original: Yelp 24.
Revised: Yelp 24.

Original:  similarity to the target set.
Revised: Nearest neighbour methods ( K NN) are used to ﬁnd examples that are semantically similar. Using sentence encoders we can search the source set D S to select the top k nearest examples by cosine similarity to the target set.

Original: We represent the target set as the mean embedding of D trainT .
Revised: We represent the target set as the mean embedding of D trainT .

Original: For question answering, where an example contains two sentences (the query and context), we refer to K NN-Q where we only encode the query text, K NN-C where we only encode the context text, or K NN-QC where we encode both concatenated together.
Revised: For question answering, where an example contains two sentences (the query and context), we refer to K NN-Q where we only encode the query text, K NN-C where we only encode the context text, or K NN-QC where we encode both concatenated together.

Original: The acquisition scoring function per example, uses either a task-speciﬁc or task-agnostic encoder E :
Revised: The acquisition scoring function per example, uses either a task-speciﬁc or task-agnostic encoder E :

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experiments are conducted on two common NLP tasks: question answering (QA) and sentiment analysis (SA), each with several available domains.
Revised: Experiments are conducted on two common NLP tasks: question answering (QA) and sentiment analysis (SA), each with several available domains.

Original: Question Answering We employ 6 diverse QA datasets from the MRQA 2019 workshop (Fisch et al., 2019), shown in Table 1 (left).
Revised: Question Answering We employ 6 diverse QA datasets from the MRQA 2019 workshop (Fisch et al., 2019), shown in Table 1 (left).

Original: 6 We sample 60k examples from each dataset for training, 5k for validation, and 5k for testing.
Revised: 6 We sample 60k examples from each dataset for training, 5k for validation, and 5k for testing.

Original: Questions and contexts are collected with varying procedures and sources, representing a wide diversity of datasets.
Revised: Questions and contexts are collected with varying procedures and sources, representing a wide diversity of datasets.

Original: Sentiment Analysis For the sentiment analysis classiﬁcation task, we follow (Blitzer et al., 2007) and (Ruder & Plank, 2018) by randomly selecting 6 Amazon multi-domain review datasets, as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011).
Revised: Sentiment Analysis For the sentiment analysis classiﬁcation task, we follow (Blitzer et al., 2007) and (Ruder & Plank, 2018) by randomly selecting 6 Amazon multi-domain review datasets, as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011).

Original: Altogether, these datasets exhibit wide diversity based on review length and topic (see Table 1).
Revised: Altogether, these datasets exhibit wide diversity based on review length and topic (see Table 1).

Original: We normalize all datasets to have 5 sentiment classes: very negative, negative, neutral, positive, and very positive.
Revised: We normalize all datasets to have 5 sentiment classes: very negative, negative, neutral, positive, and very positive.

Original: We sample 50k examples for training, 5k for validation, and 5k for testing.
Revised: We sample 50k examples for training, 5k for validation, and 5k for testing.

Original: Experimental Setup To evaluate methods for the multi-domain active learning task, we conduct the experiment described in Section 3 for each acquisition method, rotating each domain as the target set.
Revised: Experimental Setup To evaluate methods for the multi-domain active learning task, we conduct the experiment described in Section 3 for each acquisition method, rotating each domain as the target set.

Original: Model M , a BERT-Base model (Devlin et al., 2019), is chosen via hyperparameter grid search over learning rate, number of epochs, and gradient accumulation.
Revised: Model M , a BERT-Base model (Devlin et al., 2019), is chosen via hyperparameter grid search over learning rate, number of epochs, and gradient accumulation.

Original: The large volume of experiments entailed by this search space limits our capacity to benchmark performance variability due to isolated factors (the acquisition method, the target domain, or ﬁne-tuning ﬁnal models).
Revised: The large volume of experiments entailed by this search space limits our capacity to benchmark performance variability due to isolated factors (the acquisition method, the target domain, or ﬁne-tuning ﬁnal models).

Original: However, our hyperparameter search closely mimics the process of an ML practitioner looking to select a best method and model, so we believe our experiment design captures a fair comparison among methods.
Revised: However, our hyperparameter search closely mimics the process of an ML practitioner looking to select a best method and model, so we believe our experiment design captures a fair comparison among methods.

Original: See
Revised: See

Original: Algorithm 1 in Appendix Section A for full details.
Revised: Algorithm 1 in Appendix Section B for full details.

Original: C O N F ↑ C O N F ↓ E N T R ↑
Revised: C O N F ↑ C O N F ↓ E N T R ↑

Original: E N T R ↓ E N G ↑ E N G ↓ B
Revised: E N T R ↓ E N G ↑ E N G ↓ B

Original: A L D ↑ B A L D ↓ D
Revised: A L D ↑ B A L D ↓ D

Original: A L - E * D A L - T
Revised: A L - E * D A L - T

Original: * D
Revised: * D

Original: A L - E D
Revised: A L - E D

Original: A L - T
Revised: A L - T

Original: R C A R C A ~ K N N *
Revised: R C A R C A ~ K NN *

Original: K N N −−
Revised: K NN −−

Original: A b s o l u t e P e
Revised: A b s o l u t e P e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m a n c e
Revised: o r m a n c e

Original: H-Divergence RCA
Revised: H-Divergence RCA

Original: Semantic Similarity Uncertainty
Revised: Semantic Similarity Uncertainty

Original: Family
Revised: Family

Original: [Figure]
Revised: [Figure]

Original: C O N F ↑ C O N F ↓ E N T R ↑
Revised: C O N F ↑ C O N F ↓ E N T R ↑

Original: E N T R ↓ E N G ↑ E N G ↓ B
Revised: E N T R ↓ E N G ↑ E N G ↓ B

Original: A L D ↑ B A L D ↓ D
Revised: A L D ↑ B A L D ↓ D

Original: A L - E * D A L - T
Revised: A L - E * D A L - T

Original: * D
Revised: * D

Original: A L - E D
Revised: A L - E D

Original: A L - T R C
Revised: A L - T R C

Original: A R C A ~ K N N * K N N - C K N N - Q K N N - Q + C −
Revised: A R C A ~ K NN * K NN - C K NN - Q K NN - Q + C −

Original: A b s o l u t e P e
Revised: A b s o l u t e P e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m a n c e
Revised: o r m a n c e

Original: H-Divergence RCA
Revised: H-Divergence RCA

Original: Semantic Similarity Uncertainty
Revised: Semantic Similarity Uncertainty

Original: Family
Revised: Family

Original: [Figure]
Revised: [Figure]

Original:  R ESULTS 6. C OMPARING A CQUISITION M ETHODS
Revised:  R ESULTS 6. C OMPARING A CQUISITION M ETHODS

Original: Results in Figure 1 show the experiments described in Section 5: benchmarking each acquisition method for multi-domain active learning.
Revised: Results in Figure 1 show the experiments described in Section 5: benchmarking each acquisition method for multi-domain active learning.

Original: We observe for both question answering (QA) and sentiment analysis (SA), most methods manage to outperform the no-extra-labelled data baseline (0% at the y-axis) and very narrowly outperform the random selection baseline (red line).
Revised: We observe for both question answering (QA) and sentiment analysis (SA), most methods manage to outperform the no-extra-labelled data baseline (0% at the y-axis) and very narrowly outperform the random selection baseline (red line).

Original: Consistent with prior work (Lowell et al., 2019), active learning strategies in NLP have brittle and inconsistent improvements over random selection.
Revised: Consistent with prior work (Lowell et al., 2019), active learning strategies in NLP have brittle and inconsistent improvements over random selection.

Original: Our main empirical ﬁndings, described in this section, include:
Revised: Our main empirical ﬁndings, described in this section, include:

Original: • H -Divergence , and particularly DAL-E variants, consistently outperform baselines and other families of methods.
Revised: • H -Divergence , and particularly DAL-E variants, consistently outperform baselines and other families of methods.

Original: • The ordering of examples in Uncertainty methods depend signiﬁcantly on the diversity in source domains.
Revised: • The ordering of examples in Uncertainty methods depend signiﬁcantly on the diversity in source domains.

Original: BALD variants perform best among available options.
Revised: BALD variants perform best among available options.

Original: • Task-agnostic representations, used in K NN or DAL variants, providemore consistently strong results on average, but task-speciﬁc representations signiﬁcantly beneﬁt certain target sets.
Revised: • Task-agnostic representations, used in K NN or DAL variants, provide consistently strong results on average, but task-speciﬁc representations signiﬁcantly beneﬁt certain target sets.

Original: • Different families of methods rely on orthogonal notions of relevance in producing their example rankings.
Revised: • Different families of methods rely on orthogonal notions of relevance in producing their example rankings.

Original: H -Divergence methods categorically achieved the highest and most reliable scores, both as a family and individual methods, represented in the top 3 individual methods 11 / 18 times for QA, and 20 /times for SA.
Revised: H -Divergence methods categorically achieved the highest and most reliable scores, both as a family and individual methods, represented in the top 3 individual methods 11 / 18 times for QA, and 20 /times for SA.

Original: For QA, BALD ↑ and DAL-E ∗ had the best mean and median scores respectively, and for SA DAL-E achieved both the best mean and median scores.
Revised: For QA, BALD ↑ and DAL-E ∗ had the best mean and median scores respectively, and for SA DAL-E achieved both the best mean and median scores.

Original: Among these methods, our proposed
Revised: Among these methods, our proposed

Original: DAL-E variants routinely outperform DAL-T variants by a small margin on average, with equivalent training and tuning procedures.
Revised: DAL-E variants routinely outperform DAL-T variants by a small margin on average, with equivalent training and tuning procedures.

Original: We believe this is because DAL-E captures both notions of domain similarity and uncertainty.
Revised: We believe this is because DAL-E captures both notions of domain similarity and uncertainty.

Original: By design it prioritizes examples that are similar to in-domain samples, but also avoids those which are uninformative, because the model already performs well on them.
Revised: By design it prioritizes examples that are similar to in-domain samples, but also avoids those which are uninformative, because the model already performs well on them.

Original: Among Uncertainty methods , for SA methods which select  higher uncertaintyexamples vastly outperformed those which selected  low uncertainty examples.
Revised: Among Uncertainty methods , for SA methods which select for higher uncertainty vastly outperformed those which selected for low uncertainty.

Original: Interestingly, the opposite was true for QA.
Revised: The opposite is true for QA.

Original: This suggests the diversity of QA datasets contain more extreme (harmful) domain shift than the (mostly Amazon-based) SA datasets.
Revised: This suggests the diversity of QA datasets contain more extreme (harmful) domain shift than the (mostly Amazon-based) SA datasets.

Original: In both settings, the right ordering of examples with BALD ( epistemic uncertainty) achieves the best results in this family of methods, over the others, which rely on total uncertainty.
Revised: In both settings, the right ordering of examples with BALD ( epistemic uncertainty) achieves the best results in this family of methods, over the others, which rely on total uncertainty.

Original: Among Reverse Classiﬁcation Accuracy methods, our (cid:93) RCA variant also noticeably outperforms standard RCA and most other methods, aside from DAL and BALD.
Revised: Among Reverse Classiﬁcation Accuracy methods, our (cid:93) RCA variant also noticeably outperforms standard RCA and most other methods, aside from DAL and BALD.

Original: Combining (cid:93) RCA with an example ranking method is a promising direction for future work, given the performance it achieves selecting examples randomly as a Domain Budget Allocation strategy.
Revised: Combining (cid:93) RCA with an example ranking method is a promising direction for future work, given the performance it achieves selecting examples randomly as a Domain Budget Allocation strategy.

Original: Lastly, the Semantic Similarity Detection set of methods only rarely or narrowly exceed random selection.
Revised: Lastly, the Semantic Similarity Detection set of methods only rarely or narrowly exceed random selection.

Original: Intuitively, task-agnostic representations ( K NN) outperform K NN ∗ , given the task-agnostic sentence encoder was optimized for cosine similarity.
Revised: Intuitively, task-agnostic representations ( K NN) outperform K NN ∗ , given the task-agnostic sentence encoder was optimized for cosine similarity.

Original: Embedding Ablations To see the effects of embedding space on K NN and DAL, we used both a task-speciﬁc and task-agnostic embedding space.
Revised: Embedding Ablations To see the effects of embedding space on K NN and DAL, we used both a task-speciﬁc and task-agnostic embedding space.

Original: While a task-speciﬁc embedding space reduces the examples to features relevant for the task, a task-agnostic embedding space produces generic notions of similarity, unbiased by the task model.
Revised: While a task-speciﬁc embedding space reduces the examples to features relevant for the task, a task-agnostic embedding space produces generic notions of similarity, unbiased by the task model.

Original: According to Figure 1, K NN outperforms K NN ∗ .
Revised: According to Figure 1, K NN outperforms K NN ∗ .

Original: In the QA setting, K NN ∗ ’s median is below the random baseline’s.
Revised: In the QA setting, K NN ∗ ’s median is below the random baseline’s.

Original: In both plots, K NN ∗ ’s whiskers extend below 0, indicating that in some cases the method actually chooses source examples that are harmful to target domain performance.
Revised: In both plots, K NN ∗ ’s whiskers extend below 0, indicating that in some cases the method actually chooses source examples that are harmful to target domain performance.

Original: For DAL methods, task-agnostic and task-speciﬁc embeddings demonstrated mostly similar median performances.
Revised: For DAL methods, task-agnostic and task-speciﬁc embeddings demonstrated mostly similar median performances.

Original: Notably, the boxes and whiskers are typically longer for task-speciﬁc methods than task-agnostic methods.
Revised: Notably, the boxes and whiskers are typically longer for task-speciﬁc methods than task-agnostic methods.

Original: This variability indicates certain target datasets may beneﬁt signiﬁcantly from task-speciﬁc embeddings, though task-agnostic embeddings achieve more consistent results.
Revised: This variability indicates certain target datasets may beneﬁt signiﬁcantly from task-speciﬁc embeddings, though task-agnostic embeddings achieve more consistent results.

Original: Comparing Example Rankings For each setting, we quantify how similar acquisition methods rank examples from D S .
Revised: Comparing Example Rankings For each setting, we quantify how similar acquisition methods rank examples from D S .

Original: In Figure 2, for each pair of methods, we calculate the Kendall’s Tau coefﬁcient between the source example rankings chosen for a target domain, then average this coefﬁcient over the target domains.
Revised: In Figure 2, for each pair of methods, we calculate the Kendall’s Tau coefﬁcient between the source example rankings chosen for a target domain, then average this coefﬁcient over the target domains.

Original: Kendall’s Tau gives a scores [ − 1 , 1] , with -

Revised: Kendall’s Tau gives a scores [ − 1 , 1] , with -

Original: meaning perfect anti-correlation, 

Revised: meaning perfect anti-correlation,

Original: meaning no correlation, and 1 meaning perfect correlation between the rankings.
Revised: meaning no correlation, and 1 meaning perfect correlation between the rankings.

Original: Methods from different families show close to no relationship, even if they achieve similar performances, suggesting each family relies on orthogonal notions of similarity to rank example relevance.
Revised: Methods from different families show close to no relationship, even if they achieve similar performances, suggesting each family relies on orthogonal notions of similarity to rank example relevance.

Original: This suggests there is potential for combining methods from different families for this task in future work.
Revised: This suggests there is potential for combining methods from different families for this task in future work.

Original: In Sentiment tasks, all uncertainty methods had highly correlated examples, whereas in QA, ENTR had little correlation with any method.
Revised: In Sentiment tasks, all uncertainty methods had highly correlated examples. In QA, ENTR had little correlation with any method.

Original: This is likely due to the signiﬁcantly larger output space for QA models.
Revised: This is likely due to the signiﬁcantly larger output space for QA models.

Original: Compared to only 5 label classes in SA, question answering models distribute their start and end conﬁdences over sequences of up to 512, where there can be multiple valid answer candidates.
Revised: Compared to only 5 label classes in SA, question answering models distribute their start and end conﬁdences over sequences of up to 512, where there can be multiple valid answer candidates.

Original: Embedding space also largely inﬂuences the examples that methods chose.
Revised: Embedding space also largely inﬂuences the examples that methods chose.

Original: DAL methods had higher correlations with each other when they share the same embedding space; i.e. DAL-E’s ranking has a higher correlation with DAL-T than with DAL-E ∗ .
Revised: DAL methods had higher correlations with each other when they share the same embedding space; i.e. DAL-E’s ranking has a higher correlation with DAL-T than with DAL-E ∗ .

Original: 6. P ROPERTIES OF O PTIMAL E XAMPLE S ELECTION
Revised: 6. P ROPERTIES OF O PTIMAL E XAMPLE S ELECTION

Original: We examine three properties of optimally selected examples: (i) whether selecting from many diverse or one single domain leads to better performance, (ii) whether the selection of a domain or the individual examples matters more to performance, and (iii) whether selection strategies can beneﬁt from source domain information rather than treating samples as drawn from a single pool?
Revised: We examine three properties of optimally selected examples: (i) whether selecting from many diverse or one single domain leads to better performance, (ii) whether the selection of a domain or the individual examples matters more to performance, and (iii) whether selection strategies can beneﬁt from source domain information rather than treating samples as drawn from a single pool?

Original: Our ﬁndings regarding properties of optimal selection, as described in this section, include:
Revised: Our ﬁndings regarding properties of optimal selection, as described in this section, include:

Original: • Selecting a diversity of domains usually outperforms selecting examples from a single domain.
Revised: • Selecting a diversity of domains usually outperforms selecting examples from a single domain.

Original: • Acquisition functions such as DAL-E ∗ do rely on example selection, mainly to avoid the possibility of large negative outcomes.
Revised: • Acquisition functions such as DAL-E ∗ do rely on example selection, mainly to avoid the possibility of large negative outcomes.

Revised: • Domain Budget Allocation during selection may improve performance.

Revised: Surprisingly, even random selection from an “optimal” balance of domains beats our best performing acquisition methods most of the time.

Original: C O N F C O N F E N T R E
Revised: C O N F C O N F E N T R E

Original: N T R E N E R E N E R B
Revised: N T R E N E R E N E R B

Original: A L D B A L D D
Revised: A L D B A L D D

Original: A L - E * D
Revised: A L - E * D

Original: A L - E D
Revised: A L - E D

Original: A L - T * D
Revised: A L - T * D

Original: A L - T k N N * k N N ( C ) k N N - Q k N N - Q C
Revised: A L - T k N N * k N N ( C ) k N N - Q k N N - Q C

Original: C O N F C O N F
Revised: C O N F C O N F

Original: E N T R E N T R E N E R E N E R
Revised: E N T R E N T R E N E R E N E R

Original: B A L D B A L D
Revised: B A L D B A L D

Original: D A L - E
Revised: D A L - E

Original: * D A L - E
Revised: * D A L - E

Original: D A L - T * D
Revised: D A L - T * D

Original: A L - T k N N * k N N ( C ) k N N - Q k N N - Q C1.
Revised: A L - T k N N * k N N ( C ) k N N - Q k N N - Q C

Original: • Domain Budget Allocation during selection may improve performance.
Revised: [Figure]

Original: Surprisingly, even random selection from an “optimal” balance of domains beats our best performing acquisition methods most of the time.
Revised: [Figure]

Original: Are Many Diverse or One Single Domain Preferable?
Revised: Are Many Diverse or One Single Domain Preferable?

Original: To answer this question we conduct a full search over all combinations of source datasets.
Revised: To answer this question we conduct a full search over all combinations of source datasets.

Original: For each target set, we ﬁx 2k in-domain data points and sample all combinations of other source sets in 2k increments, such that altogether there are 10k training data points.
Revised: For each target set, we ﬁx 2k in-domain data points and sample all combinations of other source sets in 2k increments, such that altogether there are 10k training data points.

Original: For each combination of source sets, we conduct a simple grid search, randomly sampling the source set examples each time, and select the best model, mimicking standard practice among practitioners.
Revised: For each combination of source sets, we conduct a simple grid search, randomly sampling the source set examples each time, and select the best model, mimicking standard practice among practitioners.

Original: The result is a comprehensive search of all combinations of source sets (in 2k increments) up to 10k training points, so we can rank all combinations of domains per target, by performance.
Revised: The result is a comprehensive search of all combinations of source sets (in 2k increments) up to 10k training points, so we can rank all combinations of domains per target, by performance.

Original: Tables 2a and 2b show the optimal selections, even as discrete as 2k increments, typically select at least two, and often more domains to achieve the best performance.
Revised: Tables 2a and 2b show the optimal selections, even as discrete as 2k increments, typically select at least two or more domains to achieve the best performance.

Original: However, one of six targets for QA, orof 8 for the SA tasks achieve highest results selecting 8k examplesonly from a single domain.
Revised: However, 1 of 6 targets for QA, or 2 of 8 for the SA tasks achieve better results selecting all examples from a single domain, suggesting this is a strong baseline, if the right source domain is isolated.

Original: We also report the mean score of all permutations to demonstrate the importance of selecting the right combination of domains over a random combination of domains.
Revised: We also report the mean score of all permutations to demonstrate the importance of selecting the right set of domains over a random combination.

Original: Domains or Examples?
Revised: Domains or Examples?

Original: Which is more important, to select the right domains or the right examples within some domain?
Revised: Which is more important, to select the right domains or the right examples within some domain?

Original: From the above optimal search experiment we see selecting the right combination of domains regularly leads to strong improvements over a random combination of domains.
Revised: From the above optimal search experiment we see selecting the right combination of domains regularly leads to strong improvements over a random combination of domains.

Original: Whether example selection is more important than domain selection may vary depending on the example variety within each domain.
Revised: Whether example selection is more important than domain selection may vary depending on the example variety within each domain.

Original: We narrow our focus to how much example selection plays a role for one of the stronger acquisition functions: DAL-E ∗ .
Revised: We narrow our focus to how much example selection plays a role for one of the stronger acquisition functions: DAL-E ∗ .

Original: We ﬁx the effect of domain selection (the number of examples from each domain) but vary which examples are speciﬁcally selected.
Revised: We ﬁx the effect of domain selection (the number of examples from each domain) but vary which examples are speciﬁcally selected.

Original: Using DAL-E ∗ ’s distribution of domains, we compare the mean performance of models trained on it’s highest ranked examples against a random set of examples sampled from those same domains.
Revised: Using DAL-E ∗ ’s distribution of domains, we compare the mean performance of models trained on it’s highest ranked examples against a random set of examples sampled from those same domains.

Original: We ﬁnd a +0.46 ± 0.25% improvement for QA, and +0.12 ± 0.19% for SA.
Revised: We ﬁnd a +0.46 ± 0.25% improvement for QA, and +0.12 ± 0.19% for SA.

Original: We also compare model performances trained on random selection against the lowest ranked examples by DAL-E ∗ .
Revised: We also compare model performances trained on random selection against the lowest ranked examples by DAL-E ∗ .

Original: Interestingly, we see a -1.64 ± 0.37% performance decrease for QA, and -1.46 ± 0.56% decrease for sentiment tasks.
Revised: Interestingly, we see a -1.64 ± 0.37% performance decrease for QA, and -1.46 ± 0.56% decrease for sentiment tasks.

Original: These results suggest that example selection is an important factor beyond domain selection, especially for avoiding bad example selections.
Revised: These results suggest that example selection is an important factor beyond domain selection, especially for avoiding bad example selections.

Original: Single Pool or Domain Budget Allocation Does using information about examples’ domains during selection lead to better results than treating all examples as coming from a single unlabeled pool?
Revised: Single Pool or Domain Budget Allocation Does using information about examples’ domains during selection lead to better results than treating all examples as coming from a single unlabeled pool?

Original: Originally, we hypothesized Single Pool Strategy methods would perform better on smaller budget sizes as they add the most informative data points regardless of domain.
Revised: Originally, we hypothesized Single Pool Strategy methods would perform better on smaller budget sizes as they add the most informative data points regardless of domain.

Original: On the other hand, we thought that if the budget size is large, Domain Budget Allocation would perform best, as they choose source domains closest to the target domain.
Revised: On the other hand, we thought that if the budget size is large, Domain Budget Allocation would perform best, as they choose source domains closest to the target domain.

Original: Based on Tables 1b and 1a, we were not able to[Table] [Table] draw conclusions about this hypothesis, as each sample size n = { 8000 , 18000 , 28000 } produced roughly similar winning methods.
Revised: Based on Tables 1b and 1a, we were not able to draw conclusions about this hypothesis, as each sample size n = { 8000 , 18000 , 28000 } produced roughly similar winning methods.

Original: Future work should include a wider range of budget sizes with larger changes in method performance between sizes. 
Revised: Future work should include a wider range of budget sizes with larger changes in method performance between sizes. [Table]

Original:  In our main set of experiments, the RCA acquisition functions follow the Domain Budget Allocation strategy, while all other acquisition functions follow the Single Pool strategies.
Revised: [Table] In our main set of experiments, the RCA acquisition functions follow the Domain Budget Allocation strategy, while all other acquisition functions follow the Single Pool strategies.

Original: Based on median performance, (cid:93) RCA outperformed all other methods (we’re including BALD here due to inconsistency in performance between QA and SA) except for those in the H -Divergence family.
Revised: Based on median performance, (cid:93) RCA outperformed all other methods (we’re including BALD here due to inconsistency in performance between QA and SA) except for those in the H -Divergence family.

Original: This suggests that using domain information during selection can lead to performance gains.
Revised: This suggests that using domain information during selection can lead to performance gains.

Original: The Optimal Domain Search experiments, shown in Tables 2a and 2b, further suggest that allocating a budget from each domain can improve performance.
Revised: The Optimal Domain Search experiments, shown in Tables 2a and 2b, further suggest that allocating a budget from each domain can improve performance.

Original: For 8 out of our 14 experiments, selecting random samples according to the optimal domain distribution outperform any active learning strategy.
Revised: For 8 out of our 14 experiments, selecting random samples according to the optimal domain distribution outperform any active learning strategy.

Original: While the optimal domain distributions were not computed a priori in our experiments, this result shows the potential for Domain Budget Allocation strategies.
Revised: While the optimal domain distributions were not computed a priori in our experiments, this result shows the potential for Domain Budget Allocation strategies.

Original: Future work could reasonably improve our results by developing an acquisition function that better predicts the optimal domain distributions than (cid:93) RCA , or to even have greater performance gains by budgeting each domain, then applying an active learning strategy (e.g. DAL-E) within each budget.
Revised: Future work could reasonably improve our results by developing an acquisition function that better predicts the optimal domain distributions than (cid:93) RCA , or to even have greater performance gains by budgeting each domain, then applying an active learning strategy (e.g. DAL-E) within each budget.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We examine a challenging variant of active learning where target data is scarce, and multiple shifted domains operate as the source set of unlabeled data.
Revised: We examine a challenging variant of active learning where target data is scarce, and multiple shifted domains operate as the source set of unlabeled data.

Original: For practitioners facing multi-domain active learning, we benchmark 18 acquisition functions, demonstrating the H -Divergence family of methods and our proposed variant DAL-E achieve the best results.
Revised: For practitioners facing multi-domain active learning, we benchmark 18 acquisition functions, demonstrating the H -Divergence family of methods and our proposed variant DAL-E achieve the best results.

Original: Our analysis shows the importance of example selection in existing methods, and also the surprising potential of domain budget allocation strategies.
Revised: Our analysis shows the importance of example selection in existing methods, and also the surprising potential of domain budget allocation strategies.

Original: Combining families of methods offers a promising direction for future work in this area.
Revised: Combining families of methods, or trying domain adaptation techniques on top of selected example sets, offer promising directions for future work.

Original: AR EPRODUCIBILITY A. D ATASETS AND M ODEL T RAINING
Revised: A M ULTI -D OMAIN A CTIVE L EARNING T ASK

Revised: In this section, we would enumerate real-world settings in which a practitioner would be interested in multi-domain active learning methods.

Revised: We expect this active learning variant to be applicable to cold starts, rare classes, personalization, and settings where the modelers are constrained by privacy considerations, or a lack of labelers with domain expertise

Revised: • In the cold start scenario, for a new NLP problem, there is often little to no target data available yet (labeled or unlabelled), but there are related sources of unlabelled data to try.

Revised: Perhaps an engineer has collected small amounts of training data from an internal population.

Revised: Because the data size is small, the engineer is considering out-of-domain samples, collected from user studies, repurposed from other projects, scraped from the web, etc.. •

Revised: In the rare class scenario, take an example of a new platform/forum/social media company classifying hate speech against a certain minority group.

Revised: Perhaps the prevalence of positive, in-domain samples on the social media platform is small, so an engineer uses out-domain samples from books, other social media platforms, or from combing the internet.

Revised: • In a personalization setting, like spam ﬁltering or auto-completion on a keyboard, each user may only have a couple hundred of their own samples, but out-domain samples from other users may be available in greater quantities.

Revised: • In the privacy constrained setting, a company may collect data from internal users, user studies, and beta testers; however, a commitment to user privacy may incentivize the company to keep the amount of labeled data from the target user population low.

Revised: • Lastly, labeling in-domain data may require certain domain knowledge , which would lead to increased expenses and difﬁculty in ﬁnding annotators.

Revised: As an example, take a text classiﬁcation problem in a rare language.

Revised: It may be easy to produce out-domain samples by labeling English text and machine translating it to the rare language, whereas generating in-domain labeled data would require annotators who are ﬂuent in the rare language.

Revised: In each of these settings, target distribution data may not be amply available, but semi-similar unlabelled domains often are.

Revised: This rules out many domain adaptation methods that rely heavily on unlabelled target data.

Revised: We were able to simulate the base conditions of this problem with sentiment analysis and question answering datasets, since they are rich in domain diversity.

Revised: We believe these datasets are reasonable proxies to represent the base problem, and yield general-enough insights for a practitioner starting on this problem.

Revised: B R EPRODUCIBILITY B. D ATASETS AND M ODEL T RAINING

Revised: We choose question answering and sentiment analysis tasks as they are core NLP tasks, somewhat representative of many classiﬁcation and information-seeking problems.

Revised: Multi-domain active learning is not limited to any subset of NLP tasks, so we believe these datasets are a reasonable proxie for the problem.

Original: For question answering, the MRQA shared task (Fisch et al., 2019) includes SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA
Revised: For question answering, the MRQA shared task (Fisch et al., 2019) includes SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA

Original: (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019).
Revised: (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019).

Original: For the sentiment analysis classiﬁcation task, we use Amazon datasets following (Blitzer et al., 2007) and (Ruder & Plank, 2018), as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011).
Revised: For the sentiment analysis classiﬁcation task, we use Amazon datasets following (Blitzer et al., 2007) and (Ruder & Plank, 2018), as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011).

Original: Both question answering and sentiment analysis datasets are described in
Revised: Both question answering and sentiment analysis datasets are described in

Original: Table 1.
Revised: Table 1.

Original: For reproducibility, we share our hyper-parameter selection in Table 3.
Revised: For reproducibility, we share our hyper-parameter selection in Table 3.

Original: Hyper-parameters are taken from Longpre et al.
Revised: Hyper-parameters are taken from Longpre et al.

Original: for training all Question Answering (QA) models since their parameters are tuned for the same datasets in the MRQA Shared Task.
Revised: for training all Question Answering (QA) models since their parameters are tuned for the same datasets in the MRQA Shared Task.

Original: We found these choices to provide stable and strong results across all datasets.
Revised: We found these choices to provide stable and strong results across all datasets.

Original: For sentiment analysis, we initially experimented on a small portion of the datasets to arrive at a strong set of base hyper-parameters to tune from.
Revised: For sentiment analysis, we initially experimented on a small portion of the datasets to arrive at a strong set of base hyper-parameters to tune from.

Original: Our BERT question answering modules build upon the standard PyTorch (Paszke et al., 2019) implementations from HuggingFace, and are trained on one NVIDIA Tesla V100 GPU. 
Revised: Our BERT question answering modules build upon the standard PyTorch (Paszke et al., 2019) implementations from HuggingFace, and are trained on one NVIDIA Tesla V100 GPU. 10 .

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Figure]

Original: B E XPERIMENTAL D ESIGN
Revised: B. E XPERIMENTAL D ESIGN

Original: For more detail regarding the experimental design we include Algorith 1, using notation described in the multi-domain active learning task deﬁnition.
Revised: For more detail regarding the experimental design we include Algorith 1, using notation described in the multi-domain active learning task deﬁnition.

Original:  1: for each Acquisition Function A f do 2: for each Target set D T ∼ D do 3:
Revised: Algorithm 1 E XPERIMENTAL D ESIGN 1: for each Acquisition Function A f do 2: for each Target set D T ∼ D do 3:

Original: D trainT , D devT , D testT ∼ D T 4:
Revised: D trainT , D devT , D testT ∼ D T 4:

Original: D S :
Revised: D S :

Original: = { x ∈ D | x / ∈ D T } 5:
Revised: = { x ∈ D | x / ∈ D T } 5:

Original: M A ← T RAIN ( D trainT , D devT ) 6:
Revised: M A ← T RAIN ( D train T , D dev T ) 6:

Original: D chosen ← [ Rank x ∈ D S
Revised: D chosen ← [ Rank x ∈ D S

Original: A f ( x, M A )]
Revised: A f ( x, M A )]

Original: [: n ] 7:
Revised: [: n ] 7:

Original: D final − train =
Revised: D final − train =

Original: D trainT ∪ D chosen 8:
Revised: D trainT ∪ D chosen 8:

Original: M ← G RID S EARCH ( D final − train , D devT ) 9: ( A f , D T ) = 
Revised: M ← G RID S EARCH ( D final − train , D devT ) 9: ( A f , D T ) = s A f T ← M ( D testT ) 10: [Table]

Original: s A f T ← M ( D testT ) 10: end for 11: end for 12: return Scores Dictionary ( A f , D T )
Revised: for 11: [Table] for 12: return Scores Dictionary ( A f , D T )

Original: → s A f T
Revised: → s A f T

Original: C A CQUISITION FUNCTIONS C. T ASK A GNOSTIC E MBEDDINGS
Revised: C A CQUISITION FUNCTIONS C. T ASK A GNOSTIC E MBEDDINGS

Original: To compute the semantic similarity between two examples, we computed the example embeddings using the pre-trained model from a sentence-transformer (Reimers & Gurevych, 2019).
Revised: To compute the semantic similarity between two examples, we computed the example embeddings using the pre-trained model from a sentence-transformer (Reimers & Gurevych, 2019).

Original: We used the
Revised: We used the

Original:  RoBERTa large model, which has 24 layers, 1024 hidden layers, 16 heads, 355M parameters, and ﬁne tuning on the SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and STSBenchmark (Cer et al., 2017) datasets.
Revised: [Table] RoBERTa large model, which has 24 layers, 1024 hidden layers, 16 heads, 355M parameters, and ﬁne tuning on the SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and STSBenchmark (Cer et al., 2017) datasets.

Original: Its training procedure is documented in https://www.sbert.net/ examples/training/sts/README.html .
Revised: Its training procedure is documented in https://www.sbert.net/ examples/training/sts/README.html .

Original: C. B AYESIAN ACTIVE LEARNING BY DISAGREEMENT (BALD)
Revised: C. B AYESIAN ACTIVE LEARNING BY DISAGREEMENT (BALD)

Original: We note that Siddant and Lipton’s presentation of BALD is more closely related to the Variation
Revised: We note that Siddant and Lipton’s presentation of BALD is more closely related to the Variation

Original: Ratios acquisition function described in Gal et al.
Revised: Ratios acquisition function described in Gal et al.

Original: (2017) than the original formulation given in Houlsby et al.
Revised: (2017) than the description of dropout as a Bayesian approximation given in Gal & Ghahramani (2016).

Original: In particular, Gal et al.
Revised: In particular, Gal et al.

Original: (2017) found that Variation Ratios performed on par or better than Houlsby’s BALD on MNIST but was less suitable for ISIC2016.
Revised: (2017) found that Variation Ratios performed on par or better than Houlsby’s BALD on MNIST but was less suitable for ISIC2016.

Original: C. D ISCRIMINATIVE A CTIVE L EARNING 
Revised: C. D ISCRIMINATIVE A CTIVE L EARNING M ODEL (DAL) T RAINING

Original: [Equation] T [Equation]

Original: DAL’s training set is created using the methods detailed in Section 4.2.
Revised: DAL’s training set is created using the methods detailed in Section 4.2.

Original: The training set is then partitioned into ﬁve equally sized folds.
Revised: The training set is then partitioned into ﬁve equally sized folds.

Original: In order to predict on data that is not used to train the discriminator, we use 5-fold cross validation.
Revised: In order to predict on data that is not used to train the discriminator, we use 5-fold cross validation.

Original: The model is trained on four folds, balancing the positive and negative classes using sample weights.
Revised: The model is trained on four folds, balancing the positive and negative classes using sample weights.

Original: The classiﬁer then predicts on the single held-out fold.
Revised: The classiﬁer then predicts on the single heldout fold.

Original: This process is repeated ﬁve times so that each example is in the held out fold exactly once.
Revised: This process is repeated ﬁve times so that each example is in the held out fold exactly once.

Original: Custom model parameters are shown in Table 4; model parameters not shown in the table are the default XGBClassiﬁer parameters in xgboost 1.0.2.
Revised: Custom model parameters are shown in Table 4; model parameters not shown in the table are the default XGBClassiﬁer parameters in xgboost 1.0.2.

Original: [Table]
Revised: The motivations for choice in model and architecture are the small amount of target domain examples requiring a simple model to prevent overﬁtting and the ability of decision trees to capture collective interactions between features.

Original: D F ULL M ETHOD P ERFORMANCES
Revised: D F ULL M ETHOD P ERFORMANCES

Original: We provide a full breakdown of ﬁnal method performances in Tables 5 and 
Revised: We provide a full breakdown of ﬁnal method performances in Tables 5 and 6.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: E K ENDALL ’ S T AU  D EFINITION
Revised: E K ENDALL ’ S T AU E. D EFINITION

Original: Kendall’s Tau is a statistic that measures the rank correlation between two quantities.
Revised: Kendall’s Tau is a statistic that measures the rank correlation between two quantities.

Original: Let X and
Revised: Let X and

Original: Y be random variables with ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) as observations drawn from the joint distribution.
Revised: Y be random variables with ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) as observations drawn from the joint distribution.

Original: Given a pair ( x i , y i ) and ( x j , y j ) , where i (cid:54) =
Revised: Given a pair ( x i , y i ) and ( x j , y j ) , where i (cid:54) =

Original: j , we have:
Revised: j , we have:

Original: y j − y i x j − x i > 0 : pair is concordant y j − y i
Revised: y j − y i x j − x i > 0 : pair is concordant y j − y i

Original: x j − x i < 0 : pair is discordant y j − y
Revised: x j − x i < 0 : pair is discordant y j − y

Original: i
Revised: i

Original: x
Revised: x

Original: j − x i = 0 : pair is a tie
Revised: j − x i = 0 : pair is a tie

Original: Let n c be the number of concordant pairs and n d the number of discordant pairs.
Revised: Let n c be the number of concordant pairs and n d the number of discordant pairs.

Original: Let ties add 0.5 to the concordant and discordant pair counts each.
Revised: Let ties add 0.5 to the concordant and discordant pair counts each.

Original: Then, Kendall’s Tau is computed as:
Revised: Then, Kendall’s Tau is computed as:

Original: [Equation]
Revised: [Equation]

Revised: Here, we extend on our comparison of example rankings by presenting plots of Kendall Tau scores normalized by intra-family scores in 3.

Revised: For the sentiment setting, the ranges of intra-family Kendall

Revised: Tau coefﬁcients are smaller than the MRQA setting.

Revised: Methods in the uncertainty family have especially strong correlations with each other and much weaker with methods outside of the family.

Revised: For Hdivergence based methods, intra-family correlations are not’t as strong as for the uncertainty family;

Revised: C O N F C O N F E N T R E

Revised: N T R E N E R E N E R B

Revised: A L D B A L D D

Revised: A L - E * D

Revised: A L - E D

Revised: A L - T * D

Revised: A L - T k N N * k N N - C k N N - Q k N N - Q C

Revised: C O N F C O N F E N T R

Revised: E N T R E N E R E N E R B

Revised: A L D

Revised: B A L D D

Revised: A L - E * D A L - E

Revised: D A L - T * D

Revised: A L - T k N N * k N N - C k N N - Q k N N - Q C 1.

Revised: (a) MRQA

Revised: C O N F C O N F E N T R E

Revised: N T R E N E R E N E R B

Revised: A L D B A L D D

Revised: A L - E * D

Revised: A L - E D

Revised: A L - T * D

Revised: A L - T k N N * k N N

Revised: C O N F C O N F

Revised: E N T R E N T R

Revised: E N E R E N E R B

Revised: A L D

Revised: B A L D D

Revised: A L - E * D A L - E

Revised: D A L - T * D

Revised: A L - T 6420(b)

Revised: Sentiment in fact, the Kendall Taus between DAL-E/ K NN and DAL-T/ K NN appear to be slightly within the

Revised: H-divergence intra-family range.

Revised: Furthermore, intra-family ranges are quite large for all families in the MRQA setting.

Revised: For each method, there is at least one other method from a different family with which it had a higher Kendall

Revised: Tau coefﬁcient than the least similar methods of its own family.

Original: F R ELATING D OMAIN D ISTANCES TO P ERFORMANCE
Revised: F R ELATING D OMAIN D ISTANCES TO P ERFORMANCE

Original: We investigated why certain methods work better than others.
Revised: We investigated why certain methods work better than others.

Original: One hypothesis is that there exists a relationship between between target-source domain distances and method performance.
Revised: One hypothesis is that there exists a relationship between between target-source domain distances and method performance.

Original: We estimated the distance between two domains by computing the Wasserstein distance between random samples of 3k example embeddings from each domain.
Revised: We estimated the distance between two domains by computing the Wasserstein distance between random samples of 3k example embeddings from each domain.

Original: We experimented with two kinds of example embeddings:1.
Revised: We experimented with two kinds of example embeddings:

Original: A task agnostic embedding computed by the sentence transformer used in the K NN method, and 2.
Revised: A task agnostic embedding computed by the sentence transformer used in the K NN method, and 2.

Original: A task speciﬁc embedding computed by a model trained with the source domain used in the DAL ∗ method.
Revised: A task speciﬁc embedding computed by a model trained with the source domain used in the DAL ∗ method.

Original: Given that there are k − 1 source domains for each target domain, we tried aggregating domain distances over its mean, minimum, maximum, and variance to see if Wasserstein domain distances could be indicative of relative performance across all methods.
Revised: Given that there are k − 1 source domains for each target domain, we tried aggregating domain distances over its mean, minimum, maximum, and variance to see if Wasserstein domain distances could be indicative of relative performance across all methods.

Original: Figure3, Figure 4, Figure 5,  and Figure 6 each show, for a subset of methods, the relationship between each domain distance aggregation and the ﬁnal performance gap between the best performing method.
Revised: Figure 4, Figure 5, Figure 6, and Figure 7 each show, for a subset of methods, the relationship between each domain distance aggregation and the ﬁnal performance gap between the best performing method.

Original: Unfortunately, we found no consistent relationship for both MRQA and the sentiment classiﬁcation tasks.
Revised: Unfortunately, we found no consistent relationship for both MRQA and the sentiment classiﬁcation tasks.

Original: We believe that this result arose either because our estimated domain distances were not reliable measures of domain relevance, or because the aggregated domain distances are not independently sufﬁcient to discern relative performance differences across methods.
Revised: We believe that this result arose either because our estimated domain distances were not reliable measures of domain relevance, or because the aggregated domain distances are not independently sufﬁcient to discern relative performance differences across methods.

Original: Figures
Revised: [Figure]

Original: 3- from D T and the combined set made from choosing 3k examples from each domain in D S .

Original: Since the

Original: Wasserstein metric is symmetric, this yields k points for comparison.