Active Learning Over Multiple Domains in Natural Language Tasks
Abstract
Studies of active learning traditionally assume the target and source data stem from a single domain.
However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear a priori which data sources will help or hurt the target domain.
We survey a wide variety of techniques in active learning (AL), domain shift detection (DS), and multi-domain sampling to examine this challenging setting for question answering and sentiment analysis.
Among 18 acquisition functions from 4 families of methods, we find H -Divergence methods, and particularly our proposed variant DAL-E, yield effective results, averaging 2-3% improvements over the random baseline.
Our findings yield the first comprehensive analysis of both existing and novel methods for practitioners faced with multi-domain active learning for natural language tasks.
Introduction
New natural language problems, outside the watershed of core NLP, are often strictly limited by a dearth of labeled data.
While unlabeled data is frequently available, it is not always from the same source as the target distribution.
This is particularly prevalent for tasks characterized by (i) significant distribution shift over time, (ii) personalization for user subgroups or (iii) different collection mediums (see detailed examples discussed in Appendix A.1).
A widely-used solution to this problem is to bootstrap a larger training set using active learning (AL): a method to decide which unlabeled training examples should be labeled on a fixed annotation budget (Cohn et al., 1996; Settles, 2012).
Dor et al.
(2020); Siddhant & Lipton (2018) survey active learning methods in NLP and find notable gains over random baselines.
However, most active learning literature in NLP assumes the unlabeled source data is drawn from the same distribution as the target data (Dor et al., 2020).
This simplifying assumption avoids the frequent challenges faced by practitioners in multi-domain active learning .
In this realistic setting, there are multiple sources of data ( i.e. domains) to consider.
Where active learning baselines traditionally select examples the model is least confident on (Settles, 2009), in this setting it could lead to distracting examples from very dissimilar distributions. [Table]
A few previous works investigated active learning under distribution shifts, though mainly in image classification, with single source and target domains.
Kirsch et al.
(2021) finds that BALD, which is often considered the state of the art for unshifted domain settings, can get stuck on irrelevant source domain or junk data.
Zhao et al.
(2021) and Saha et al.
(2011) address label and covariate shift respectively, but not general distribution shift.
In this work we empirically examine four separate families of methods (uncertainty-based, HDivergence, reverse classification accuracy, and semantic similarity detection) over several question answering and sentiment analysis datasets, following (Lowell et al., 2019; Elsahar & Gallé, 2019), to provide actionable insights to practitioners facing this challenging variant of active learning for natural language.
While previous work has investigated similar settings (Saha et al., 2011; Liu et al., 2015; Zhao et al., 2021; Kirsch et al., 2021;
He et al., 2021) we contribute, to our knowledge, the first rigorous formalization and broad survey of methods within NLP.
We find that many families of techniques for active learning and domain shift detection fail to reliably beat random baselines in this challenging variant of active learning, but certain H -Divergence methods are consistently strong.
Our analysis identifies stark dissimilarities of these methods’ example selection, and suggests domain diversity is an important factor in achieving strong results.
Multi-Domain Active Learning
Suppose we have multiple domains D 1 , D 2 , ..., D k .
Let one of the k domains be the target set
D T , and let the other k − 1 domains comprise the source set D S
= (cid:83) i ̸ =
T D i .
Assume we have small samples of labeled data points ( x, y ) from D T ( D trainT , D devT , D testT ∼ D T . 4 ) and a large sample of unlabeled points ( x ) from the source domains ( D S =
(cid:83) i ̸ =
T D i ).
We have the following task:
Choose n samples from D S to label.
D chosenS ⊂ D S , | D chosenS | = n , selected by argmax x ∈ D S
A f ( x ) where A f is an acquisition function: a policy to select unlabeled examples from D S for labeling.
Train a model M on D final − train , validating on D devT .
D final − train =
D trainT ∪ D chosenS 3.
Evaluate M on D testT , giving score s .
For Step 1, the practitioner chooses n samples with the highest scores according to their acquisition function A f .
M is fine-tuned on these n samples, then evaluated on D testT to demonstrate A f ’s ability to choose relevant out-of-distribution training examples.
Methods
We identify four families of methods relevant to active learning over multiple shifted domains:
Uncertainty methods , H -Divergence techniques, Semantic Similarity Detection , and Reverse
Classification Accuracy .
We derive ∼ 18 active learning variants, comprising the most prevalent and effective from prior work, and novel variants of existing paradigms for the multi-domain active learning setting (see K NN, (cid:93) RCA and DAL-E).
Uncertainty methods are common in standard active learning for measuring example uncertainty or familiarity to a model.
Our uncertainty baselines consist of Confidence (CONF) , which scores candidates based on the maximum value in the softmax output; Entropy (ENTR) , which calculates the entropy over the softmax output; Energy (ENG) , which calculates an energy score over the output logits, and is less susceptible to overconfidence issues of softmax approaches; and Bayesian Active
Learning by Disagreement (BALD) measures prediction disagreement over multiple inference passes with dropout.
We break each uncertainty method into two sub-methods: the first chooses samples in ascending order of score and the second in descending order.
See A.3.1 for acquisition functions.
H -Divergence techniques train classifiers for domain shift detection.
Ben-David et al.
(2006, 2010) formalize the divergence between two domains as the H -Divergence, which they approximate as the difficulty for a discriminator to differentiate between the two.
We explore two variants of
Discriminative Active Learning (DAL), which applies this concept to the active learning setting (Gissin & Shalev-Shwartz, 2019) 6 .
The first, Discriminative Active Learning - Target (DAL-T) , trains a discriminator to distinguish between target and source examples and selects the source examples that the discriminator most confused with the target.
The second, Discriminative Active
Learning - Error (DAL-E) , which is a novel variant, trains a discriminator to distinguish between source examples and erroneous target examples, selecting source samples that were most confused with erroneous target samples.
See Appendix A.3.2 for more details.
Semantic Similarity Detection finds data points similar to points in the target domain.
Nearest neighbour methods ( K NN) are used to find examples that are semantically similar.
Using sentence encoders we can search the source set D S to select the top k nearest examples by cosine similarity to the target set, which is represented as the mean embedding of D trainT .
See Appendix A.3.4 for more details.
Reverse Classification Accuracy (RCA) estimates how effective a source set is as a training data for target test set D T (Fan & Davidson, 2006; Elsahar & Gallé, 2019).
This involves pseudo-labeling source examples to train child models.
Then, for active learning, we randomly sample from the domain that trained the best performing child model.
Standard RCA only selects examples from one domain.
We develop a novel variant which samples from multiple domains, proportional to their relative performance on the target domain.
See Appendix A.3.3 for more details.
Experiments
Experiments are conducted on two common NLP tasks: question answering (QA) and sentiment analysis (SA), each with several available domains.
See Appendix A.2.1 for details about the datasets.
To evaluate methods for the multi-domain active learning task, we conduct the experiment described in Section 2 for each acquisition method, rotating each domain as the target set.
Model M , a BERTBase model (Devlin et al., 2019), is chosen via hyperparameter grid search over learning rate, number of epochs, and gradient accumulation.
See Algorithm 1 in Appendix Section A.2 for full details.
Results
Results can be seen in Figure 1.
We observe for both question answering (QA) and sentiment analysis (SA), most methods manage to outperform the no-extra-labelled data baseline (0% at the y-axis) and very narrowly outperform the random selection baseline (red line).
Consistent with prior work (Lowell et al., 2019), active learning strategies in NLP have brittle and inconsistent improvements over random selection.
H -Divergence methods categorically achieved the highest and most reliable scores, both as a family and individual methods, represented in the top 3 individual methods 11 / 18 times for QA, and 20 / 24 times for SA.
For QA, BALD ↑ and DAL-E ∗ had the best mean and median scores respectively, and for SA DAL-E achieved both the best mean and median scores.
Among these methods, our proposed DAL-E variants routinely outperform DAL-T variants by a small margin on average.
We believe this is because DAL-E captures both notions of domain similarity and uncertainty.
By design it prioritizes examples that are similar to in-domain samples, but also avoids those which the model already performs well on.
Among Uncertainty methods , for SA methods which select for higher uncertainty vastly outperformed those which selected for low uncertainty.
The opposite is true for QA.
This suggests the diversity of QA datasets contain more extreme (harmful) domain shift than the (mostly Amazon-based)
SA datasets.
In both settings, the right ordering of examples with BALD ( epistemic uncertainty) achieves the best results in this family of methods, over the others, which rely on total uncertainty.
C O N F ↑ C O N F ↓ E N T R ↑
E N T R ↓ E N G ↑ E N G ↓ B
A L D ↑ B A L D ↓ D
A L - E * D A L - T
* D
A L - E D
A L - T
R C A R C A ~ K NN *
K NN −−
A b s o l u t e P e
r
f
o r m a n c e
H-Divergence RCA
Semantic Similarity Uncertainty
Family
[Figure]
C O N F ↑ C O N F ↓ E N T R ↑
E N T R ↓ E N G ↑ E N G ↓ B
A L D ↑ B A L D ↓ D
A L - E * D A L - T
* D
A L - E D
A L - T R C
A R C A ~ K NN * K NN - C K NN - Q K NN - Q + C −
A b s o l u t e P e
r
f
o r m a n c e
H-Divergence RCA
Semantic Similarity Uncertainty
Family
[Figure]
Among Reverse Classification Accuracy methods, our (cid:93) RCA variant also noticeably outperforms standard RCA and most other methods, aside from DAL and BALD.
Combining (cid:93) RCA with an example ranking method is a promising direction for future work, given the performance it achieves selecting examples randomly as a Domain Budget Allocation strategy.
Lastly, the Semantic Similarity Detection set of methods only rarely or narrowly exceed random selection.
Intuitively, task-agnostic representations ( K NN) outperform K NN ∗ , given the task-agnostic sentence encoder was optimized for cosine similarity.
Embedding Ablations To see the effects of embedding space on K NN and DAL, we used both a task-specific and task-agnostic embedding space.
While a task-specific embedding space reduces the examples to features relevant for the task, a task-agnostic embedding space produces generic notions of similarity, unbiased by the task model.
According to Figure 1, K NN outperforms K NN ∗ .
In the QA setting, K NN ∗ ’s median is below the random baseline’s.
In both plots, K NN ∗ ’s whiskers extend below 0, indicating that in some cases the method actually chooses source examples that are harmful to target domain performance.
For DAL methods, task-agnostic and task-specific embeddings demonstrated mostly similar median performances.
Notably, the boxes and whiskers are typically longer for task-specific methods than task-agnostic methods.
This variability indicates certain target datasets may benefit significantly from task-specific embeddings, though task-agnostic embeddings achieve more consistent results.
Further Experiments Further experimental work can be seen in the Appendix.
In Appendix
A.5, we examine example rankings and find that different families show close to no relationship, suggesting that families rely on orthogonal notions of similarity to rank examples.
In Appendix A.5.3, we explore questions relating to (i) selecting from many or one domain, (ii) selecting a whole domain or individual examples, and (iii) treating the pool as one single set versus breaking it into respective domains.
Conclusion
We examine a challenging variant of active learning where target data is scarce, and multiple shifted domains operate as the source unlabeled dataset.
For practitioners facing multi-domain active learning, we benchmark 18 acquisition functions, demonstrating the H -Divergence family of methods and our variant DAL-E achieve best results.
Our analysis shows the importance of example selection in existing methods.
Combining families of methods, or trying domain adaptation techniques on top of selected example sets, offer promising directions for future work.
Acknowledgments and Disclosure of Funding
The authors would like to thank Moises Goldszmidt, Ehsan Mousavi and Stephen Pulman for helpful discussions and feedback.
A Appendix A.1 Real World Settings
In this section we enumerate real-world settings in which a practitioner could be interested in multi-domain active learning methods.
We expect multi-domain active learning to be applicable to cold starts, rare classes, personalization and settings where the modelers are constrained by privacy considerations or a lack of labelers with domain expertise.
• In the cold start scenario, for a new NLP problem, there is often little to no target data available yet (labeled or unlabelled), but there are related sources of unlabelled data to try.
Perhaps an engineer has collected small amounts of training data from an internal population.
Because the data size is small, the engineer is considering out-of-domain samples, collected from user studies, repurposed from
other projects, scraped from the web, etc..
• In the rare class scenario, take an example of a new platform/forum/social media company classifying hate speech against a certain minority group.
Perhaps the prevalence of positive, in-domain samples on the social media platform is small, so an engineer uses out-domain samples from books, other social media platforms, or from combing the internet.
• In a personalization setting, like spam filtering or auto-completion on a keyboard, each user may only have a couple hundred of their own samples, but out-domain samples from other users may be available in greater quantities.
• In the privacy constrained setting, a company may collect data from internal users, user studies, and beta testers; however, a commitment to user privacy may incentivize the company to keep the amount of labeled data from the target user population low.
• Lastly, labeling in-domain data may require certain domain knowledge , which would lead to increased expenses and difficulty in finding annotators.
As an example, take a text classification problem in a rare language.
It may be easy to produce out-domain samples by labeling English text and machine translating it to the rare language, whereas generating in-domain labeled data would require annotators who are fluent in the rare language.
In each of these settings, target distribution data may not be amply available, but semi-similar unlabelled domains often are.
This rules out many domain adaptation methods that rely heavily on unlabelled target data.
We were able to simulate the base conditions of this problem with sentiment analysis and question answering datasets, since they are rich in domain diversity.
We believe these datasets are reasonable proxies to represent the base problem, and yield general-enough insights for a practitioner starting on this problem.
A.2 Reproducibility A.2.1 Datasets
We choose question answering and sentiment analysis tasks as they are core NLP tasks, somewhat representative of many classification and information-seeking problems.
Multi-domain active learning is not limited to any subset of NLP tasks, so we believe these datasets are a reasonable proxy for the problem.
For question answering, we employ 6 diverse QA datasets from the MRQA 2019 workshop (Fisch et al., 2019), which includes SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016),
TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), and
Natural Questions (Kwiatkowski et al., 2019).
8 We sample 60k examples from each dataset for training, 5k for validation, and 5k for testing.
Questions and contexts are collected with varying procedures and sources, representing a wide diversity of datasets.
For the sentiment analysis classification task, we use Amazon datasets following (Blitzer et al., 2007) and (Ruder & Plank, 2018) by randomly selecting 6 Amazon multi-domain review datasets, as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011).
Altogether, these datasets exhibit wide diversity based on review length and topic (see Table 1).
We normalize all datasets to have 5 sentiment classes: very negative, negative, neutral, positive, and very positive.
We sample 50k examples for training, 5k for validation, and 5k for testing.
Both question answering and sentiment analysis datasets are described in Table 1.
A.2.2 Model Training
For reproducibility, we share our hyper-parameter selection in Table 2.
Hyper-parameters are taken from Longpre et al.
for training all Question Answering (QA) models since their parameters are tuned for the same datasets in the MRQA Shared Task.
We found these choices to provide stable and strong results across all datasets.
For sentiment analysis, we initially experimented on a small portion of the datasets to arrive at a strong set of base hyper-parameters to tune from. [Table]
Our BERT question answering modules build upon the standard PyTorch (Paszke et al., 2019) implementations from HuggingFace [Table] and are trained on one NVIDIA Tesla V [Table]
[Table]
A.2.3 Experimental Design
For more detail regarding the experimental design we include Algorithm 1, using notation described in the multi-domain active learning task definition.
A.3 Method Details
In this section, we describe each method in more detail.
Each method produces a full ranking of examples in the source set D S . To rank examples, most acquisition methods train an acquisition model, M A , using the same model architecture as M . M A is trained on all samples from D trainT , except for DAL and K NN, which split D trainT into two equal segments, one for training M A and one for an internal model. Some methods have both ascending and descending orders of these rankings
1: for each Acquisition Function A f do 2: for each Target set D T ∼ D do 3:
D trainT , D devT , D testT ∼ D T 4:
D S :
= { x ∈ D | x / ∈ D T } 5:
M A ← T RAIN ( D trainT , D devT ) 6:
D chosen ← [ Rank x ∈ D S
A f ( x, M A )]
[: n ] 7:
D final − train =
D trainT ∪ D chosen 8:
M ← G RID S EARCH ( D final − train , D devT ) 9: ( A f , D T ) =
s A f T ← M ( D testT ) 10: end for 11: end for 12: return Scores Dictionary ( A f , D T )
→ s A f T (denoted by ↑ and ↓ respectively, in the method abbreviations), to test whether similar or distant examples are preferred in a multi-domain setting.
Certain methods use vector representations of candidate examples.
We benchmark with both taskagnostic and task-specific encoders.
The task-agnostic embeddings are taken from the last layer’s CLS token in Reimers & Gurevych (2019)’s sentence encoder (Appendix for details).
The task-specific embeddings are taken from the last layer’s
CLS token in the trained model M A .
Let E be the encoder function.
The motivation of the task-specific variant is that each example’s representation will capture taskrelevant differences between examples while ignoring irrelevant differences.
The versions of DAL and K NN methods that use task-specific vectors are denoted with “ ∗ ” in their abbreviation.
Otherwise, they use task-agnostic vectors.
A.3.1 Uncertainty Method Acquisition Functions
Let Y be the set of all possible labels produced from the model M ( x ) and l y be the logit value for y ∈ Y .
Let P ( y | x ) be the probabilities from the softmax output.
CONF has an acquisition function:
[Equation]
and ENTR an an acquisition function:
[Equation]
For ENG , the acquisition function is:
[Equation]
where we chose T = 1 to maximize the distinction between in- and out-domain predictions.
For BALD , we conduct T = 20 forward passes on x .
ˆ y t
= argmax i
P
( y i | x ) t , representing the predicted class on the t -th model pass on x .
Following (Lowell et al., 2019), ties are broken by taking the mean label entropy over all T runs.
The acquisition function is:
[Equation]
We note that Siddant and Lipton’s presentation of BALD is more closely related to the Variation
Ratios acquisition function described in Gal et al.
(2017) than the description of dropout as a Bayesian approximation given in Gal & Ghahramani (2016).
In particular, Gal et al.
(2017) found that
Variation Ratios performed on par or better than Houlsby’s BALD on MNIST but was less suitable for ISIC2016.
A.3.2 Discriminative Active Learning Model (DAL)
For the following methods, let D train − B T be the 1k examples from D trainT that were not used to train
M A .
We use samples both from D train − B T and D S to train the discriminator, g .
We assign samples origin labels l , which depend on the DAL variant.
Samples from D S with discriminator predictions closest to 1 are selected for labeling.
The acquisition scoring function for each DAL method and training set definition, respectively, are:
[Equation]
DAL-T trains a discriminator g to distinguish between target examples in D train − B T and out-ofdistribution examples from D S .
For DAL-T, l = 1 D train − B T ( x ) .
DAL-E is a novel variant of DAL.
DAL-E’s approach is to find examples that are similar to those in the target domain that M A misclassified.
We partition D train − B T further into erroneous samples
D errT and correct samples D corrT , where D train − B T
[Equation] corrT .
For DAL-E, [Equation] = 1 D errT ( x ) .
We use an XGBoost decision tree (Chen & Guestrin, 2016) as DAL’s discriminator model.
DAL’s training set is created using the methods detailed in Section ?? .
The training set is then partitioned into five equally sized folds.
In order to predict on data that is not used to train the discriminator, we use 5-fold cross validation.
The model is trained on four folds, balancing the positive and negative classes using sample weights.
The classifier then predicts on the single held-out fold.
This process is repeated five times so that each example is in the held out fold exactly once.
Custom model parameters are shown in Table 3; model parameters not shown in the table are the default XGBClassifier parameters in xgboost 1.0.2.
The motivations for choice in model and architecture are the small amount of target domain examples requiring a simple model to prevent overfitting and the ability of decision trees to capture collective interactions between features.
A.3.3 Reverse Classification Accuracy
Reverse Classification Accuracy (RCA) estimates how effective source set D i,i ∈ S is as a training data for target test set D T (Fan & Davidson, 2006; Elsahar & Gallé, 2019).
Without gold labels for
D i we compute soft labels instead, using the BERT-Base M A trained on the small labeled set D trainT .
We then train a child model M i on D i using these soft labels, and evaluate the child model on D devT .
RCA chooses examples randomly from whichever domain i produced the highest score
s i .
[Equation]
Standard RCA only selects examples from one domain D i .
We develop a novel variant which samples from multiple domains, proportional to their relative performance on the target domain D devT .
RCAsmoothed ( (cid:93) RCA ) selects | D choseni | examples from source domain i , based on the relative difference between the performance
s i (of child model M i trained on domain i with pseudo-labels from M A ) on the target domain, and the performance s T of a model trained directly on the target domain D devT .
Since these strategies directly estimates model performance on the target domain resulting from training on each source domain, RCA and (cid:93) RCA are strong Domain Budget Allocation candidates.
A.3.4 Nearest Neighbour / Semantic Similarity
For question answering, where an example contains two sentences (the query and context), we refer to K NN-Q where we only encode the query text, K NN-C where we only encode the context text, or K NN-QC where we encode both concatenated together.
The acquisition scoring function per example is:
[Equation]
[Table]
[Table]
5 Task Agnostic Embeddings
To compute the semantic similarity between two examples, we computed the example embeddings using the pre-trained model from a sentence-transformer (Reimers & Gurevych, 2019).
We used the
RoBERTa large model, which has 24 layers, 1024 hidden layers, 16 heads, 355M parameters, and fine tuning on the SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and STSBenchmark (Cer et al., 2017) datasets.
Its training procedure is documented in https://www.sbert.net/ examples/training/sts/README.html .
A.4 Full Method Performances
We provide a full breakdown of final method performances in Tables 4 and 5.
A.4.1 Effect of Sample Size on Strategy
Originally, we hypothesized Single Pool Strategy methods would perform better on smaller budget sizes as they add the most informative data points regardless of domain.
On the other hand, we thought that if the budget size is large, Domain Budget Allocation would perform best, as they choose source domains closest to the target domain.
Based on Figures 1b and 1a, we were not able to draw conclusions about this hypothesis, as each sample size n = { 8000 , 18000 , 28000 } produced roughly similar winning methods.
Future work should include a wider range of budget sizes with larger changes in method performance between sizes.
[Table]
67.96 A MZN -T 66.
69.62 68.1 68.8 67.2 69.72 I MDB 59.
60.6 60.32 60.4 60.64 Y ELP 66.
C O N F C O N F E N T R E
N T R E N E R E N E R B
A L D B A L D D
A L - E * D
A L - E D
A L - T * D
A L - T k N N * k N N ( C ) k N N - Q k N N - Q C
C O N F C O N F
E N T R E N T R E N E R E N E R
B A L D B A L D
D A L - E
* D A L - E
D A L - T * D
A L - T k N N * k N N ( C ) k N N - Q k N N - Q C 1. means an inverse relationship.
A.5 Comparing Example Rankings
For each setting, we quantify how similar acquisition methods rank examples from D S .
In Figure 2, for each pair of methods, we calculate the Kendall’s Tau coefficient between the source example rankings chosen for a target domain, then average this coefficient over the target domains.
Kendall’s
Tau gives a scores [ − 1 , 1] , with -1 meaning perfect anti-correlation,
meaning no correlation, andmeaning perfect correlation between the rankings.
Methods from different families show close to no relationship, even if they achieve similar performances, suggesting each family relies on orthogonal notions of similarity to rank example relevance.
This suggests there is potential for combining methods from different families in future work.
In Sentiment tasks, all uncertainty methods had highly correlated examples.
In QA, ENTR had little correlation with any method.
This is likely due to the significantly larger output space for QA models.
Compared to only 5 label classes in SA, question answering models distribute their start and end confidences over sequences of up to 512, with multiple feasible answer candidates.
Embedding space also largely influences the examples that methods chose.
DAL methods had higher correlations with each other when they share the same embedding space; i.e. DAL-E’s ranking has a higher correlation with DAL-T than with DAL-E ∗ .
A.5.1 Kendall’s Tau Definition
Kendall’s Tau is a statistic that measures the rank correlation between two quantities.
Let X and
Y be random variables with ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) as observations drawn from the joint distribution.
Given a pair ( x i , y i ) and ( x j , y j ) , where i ̸
= j , we have:
[Figure]
: pair is concordant [Figure] : pair is discordant
[Figure]
= 0 : pair is a tie
Let n c be the number of concordant pairs and n d the number of discordant pairs.
Let ties add 0.5 to the concordant and discordant pair counts each.
Then, Kendall’s Tau is computed [-as: 
-]as:τ = n c − n d n c + n d
[Figure]
- Q C
C O N F
C O N F
E N T R E N T R
E N E R E N E R
B A L D B A L D
D A L - E *
D A L - E
D A L - T *
D
A L - T k N N * k N N - C k N N - Q k N N - Q C 1.
(a) MRQA
C O N F C O N F E N T R E
N T R E N E R E N E R
B A L D
B A L D D
A L - E * D
A L - E D
A L - T * D
A L - T k N N * k N N
C O N F C O N F
E N T R E N T R
E N E R E N E R
B A L D B A L D
D A L - E *
D A L - E
D A L - T *
D A L - T
[Figure]
cell’s corresponding Kendall Tau score is within the intra-family range, it’s value will be in [0 , 1] .
Below the range is negative, and above the range is greater than 1. A.5.2 Inter-Family Comparison Here, we extend on our comparison of example rankings by presenting plots of Kendall Tau scores normalized by intra-family scores in Figure 3. For the sentiment setting, the ranges of intra-family
Kendall Tau coefficients are smaller than the MRQA setting.
Methods in the uncertainty family have especially strong correlations with each other and much weaker with methods outside of the family.
For H-divergence based methods, intra-family correlations are not’t as strong as for the uncertainty family; in fact, the Kendall Taus between DAL-E/ K NN and DAL-T/ K NN appear to be slightly within the H-divergence intra-family range.
Furthermore, intra-family ranges are quite large for all families in the MRQA setting.
For each method, there is at least one other method from a different family with which it had a higher Kendall
Tau coefficient than the least similar methods of its own family.
A.5.3 Properties of Optimal Example Selection
We examine three properties of optimally selected examples: (i) whether selecting from many diverse or one single domain leads to better performance, (ii) whether the selection of a domain or individual examples matters more to performance, and (iii) whether selection strategies can benefit from source domain information rather than treating samples as drawn from a single pool?
Our findings regarding properties of optimal selection include:
• Selecting a diversity of domains usually outperforms selecting examples from a single domain.
• Acquisition functions such as DAL-E ∗ do rely on example selection, mainly to avoid the possibility of large negative outcomes.
• Allocating a budget for each domain may improve performance.
Surprisingly, even random selection from an “optimal” balance of domains beats our best performing acquisition methods most of the time.
Are Many Diverse or One Single Domain Preferable?
To answer this question we conduct a full search over all combinations of source datasets.
For each target set, we fix 2k in-domain data points and sample all combinations of other source sets in 2k increments, such that altogether there are 10k training data points.
For each combination of source sets, we conduct a simple grid search, randomly sampling the source set examples each time, and select the best model, mimicking standard practice among practitioners.
The result is a comprehensive search of all combinations of source sets (in 2k increments) up to 10k training points, so we can rank all combinations of domains per target, by performance.
Tables 6a and 6b show the optimal selections, even as discrete as 2k increments, typically select at least two or more domains to achieve the best performance.
However, 1 of 6 targets for QA, or 2 of 8 for the SA tasks achieve better results selecting all examples from a single domain, suggesting this is a strong baseline, if the right source domain is isolated.
We also report the mean score of all permutations to demonstrate the importance of selecting the right set of domains over a random combination.
Domains or Examples?
Which is more important, to select the right domains or the right examples within some domain?
From the above optimal search experiment we see selecting the right combination of domains regularly leads to strong improvements over a random combination of domains.
Whether example selection is more important than domain selection may vary depending on the example variety within domains.
We narrow our focus to how much example selection plays a role for one of the stronger acquisition functions: DAL-E ∗ .
We fix the effect of domain selection (the number of examples from each domain) but vary which examples are specifically selected.
Using DAL-E ∗ ’s distribution of domains, we compare the mean performance of models trained on it’s highest ranked examples against a random set of examples sampled from those same domains.
We find a +0.46 ± 0.25% improvement for QA, and +0.12 ± 0.19% for SA.
We also compare model performances trained on random selection against the lowest ranked examples by DAL-E ∗ .
Interestingly, we see a -1.64 ± 0.37% performance decrease for QA, and -1.46 ± 0.56% decrease for sentiment tasks.
These results suggest that example selection is an important factor beyond domain selection, especially for avoiding bad example selections.
Single Pool or Domain Budget Allocation Does using information about examples’ domains during selection lead to better results than treating all examples as coming from a single unlabeled pool?
In our main set of experiments, the RCA acquisition functions follow the Domain Budget
Allocation strategy, which allocates an example budget for each domain.
All other acquisition functions follow Single Pool strategies, which treat all source samples as coming from a single pool.
Based on median performance, (cid:93) RCA outperformed all other methods (we’re including BALD here due to inconsistency in performance between QA and SA) except for those in the H -Divergence family.
This suggests that using domain information during selection can lead to performance gains.
The Optimal Domain Search experiments, shown in Tables 6a and 6b, further suggest that allocating a budget from each domain can improve performance.
For 8 out of our 14 experiments, selecting [Table] [Table] random samples according to the optimal domain distribution outperform any active learning strategy.
While the optimal domain distributions were not computed a priori in our experiments, this result shows the potential for Domain Budget Allocation strategies.
Future work could reasonably improve our results by developing an acquisition function that better predicts the optimal domain distributions than (cid:93) RCA , or to even have greater performance gains by budgeting each domain, then applying an active learning strategy (e.g. DAL-E) within each budget.
A.6 Relating Domain Distances to Performance
We investigated why certain methods work better than others.
One hypothesis is that there exists a relationship between between target-source domain distances and method performance.
We estimated the distance between two domains by computing the Wasserstein distance between random samples of 3k example embeddings from each domain.
We experimented with two kinds of example embeddings:
A task agnostic embedding computed by the sentence transformer used in the K NN method, and 2.
A task specific embedding computed by a model trained with the source domain used in the DAL ∗ method.
Given that there are k − 1 source domains for each target domain, we tried aggregating domain distances over its mean, minimum, maximum, and variance to see if Wasserstein domain distances could be indicative of relative performance across all methods.
Figure 4, Figure 5, Figure 6, and Figure 7 each show, for a subset of methods, the relationship between each domain distance aggregation and the final performance gap between the best performing method.
Unfortunately, we found no consistent relationship for both MRQA and the sentiment classification tasks.
We believe that this result arose either because our estimated domain distances were not reliable measures of domain relevance, or because the aggregated domain distances are not independently sufficient to discern relative performance differences across methods.