On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
Abstract
Many pairwise classiﬁcation tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., 99 .
99% of examples are negatives).
In contrast, many recent datasets heuristically choose examples to ensure label balance.
We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only 2 .
4% average precision when evaluated on realistically imbalanced test data.
We instead collect training data with active learning, using a BERT-based embedding model to efﬁciently retrieve uncertain points from a very large pool of unlabeled utterance pairs.
By creating balanced training data with more informative negative examples, active learning greatly improves average precision to 32 .
5% on QQP and 20 .
1% on WikiQA.
Introduction
For most pairwise classiﬁcation tasks in NLP, the most realistic data distribution has extreme label imbalance (e.g., 99 .
99% of examples have the same label).
In question deduplication (Iyer et al., 2017), the vast majority of pairs of questions from an online forum are not duplicates.
In open-domain question answering (Yang et al., 2015; Lee et al., 2019), almost any randomly sampled document will not answer a given question.
Random pairs of sentences from a diverse distribution will have no relation between them in natural language inference (Bowman et al., 2015), as opposed to an entailment or contradiction relationship.
While past work has recognized the importance of label imbalance in NLP (Lewis et al., 2004; Chawla et al., 2004), many recently released datasets are heuristically collected to ensure label balance, generally for ease of training.
For instance,
[Figure]
the Quora Question Pairs (QQP) dataset (Iyer et al., 2017) was generated by mining non-duplicate questions that were heuristically determined to be nearduplicates.
The SNLI dataset had crowdworkers generate inputs to match a speciﬁed label distribution (Bowman et al., 2015).
In this work, we show that models trained on heuristically balanced datasets deal poorly with natural label imbalance at test time: They have very low average precision on realistically imbalanced test data created by taking all pairs of test utterances.
Instead of heuristically producing static training datasets, we study adaptive data collection methods.
In particular, we apply the pool-based active learning framework (Settles, 2009) to extremely imbalanced pairwise tasks.
At training time, the system has query access (i.e., the ability to collect a limited subset of the labels) to a large unlabeled dataset formed by taking all utterance pairs from a set of training utterances.
For example, in the question deduplication setting, we might have a budget to annotate pairs of questions as “duplicate” or “not duplicate,” and wish to train a model on this data to ﬁnd new duplicate pairs with high precision.
Data collection for extremely imbalanced pairwise tasks is challenging because the pool of unlabeled examples is very large (as it grows quadratically in the number of utterances) and very few of the examples are positive.
We collect balanced training data using uncertainty sampling, an adaptive method that queries labels for examples on which a model trained on previously queried data has high uncertainty (Lewis and Gale, 1994).
To lower the computational cost of searching for uncertain points, we propose combining active learning with a BERT embedding model for which uncertain points can be located efﬁciently using nearest neighbor search.
In this work, we empirically show that our use of adaptive data collection yields signiﬁcant gains over static heuristics.
In order to compare methods without collecting data separately for each method and each run, we perform retrospective data collection with imputed labels to simulate data collection.
Uncertainty sampling with our BERT embedding model achieves 32 .
5% and 20 .
1% average precision for QQP and WikiQA, respectively.
In contrast, state-of-the-art models trained on the original heuristically collected data each have only average precision of 2 . 4% .
Setting
The pairwise tasks described above fall under a more general category of binary classiﬁcation tasks, those with an input space X and output space { 0 , 1 } .
We assume the label y is a deterministic function of x , which we write y ( x ) .
A classiﬁcation model p θ yields probability estimates p θ ( y | x ) where x ∈ X .
Our setting has two aspects: The way training data is collected via label queries (Section 2.1) and the way we evaluate the model p θ ( y | x ) by measuring average precision (Section 2.2).
This work focuses on pairwise tasks (Section 2.3), which enables efﬁcient active learning (Section 4).
2. Data collection
In our setting, a system is given an unlabeled dataset D trainall ⊆
X .
The system can query an input x ∈ D train all and receive the corresponding label y ( x ) .
The system is given a budget of n queries to build a labeled training dataset of size n .
2. Evaluation
Following standard practice for imbalanced tasks, we evaluate on precision, recall, and average precision (Lewis, 1995; Manning et al., 2008).
A scoring function S : X → R (e.g., S ( x ) =
p θ ( y = 1 | x ) ) is used to rank examples x , where an ideal S assigns all positive examples { x : y ( x ) = 1 } a higher score than all negative examples { x : y ( x ) = 0 } .
Given a test dataset D testall ⊆
X , deﬁne the number of true positives, false positives, and false negatives of a scoring function S at a threshold γ as:
[Equation]
For any threshold γ , deﬁne the precision P ( S, γ ) and recall R ( S, γ ) of a scoring function S as:
[Equation]
Let Γ = { S ( x ) : x ∈ D testall } be the set of scores of the dataset.
By sweeping over all distinct values Γ in descending order, we trace out the precisionrecall curve.
The area under the precision recall curve or average precision (AP) is deﬁned as:
[Equation]
where γ 0 = ∞ and γ 1 > γ 2 > .
γ | Γ | and γ i ∈ Γ .
Note that high precision requires very high accuracy when the task is extremely imbalanced.
For example, if only one in 10 , 000 examples in D testall is positive and 50% precision at some recall is achieved, that implies 99.99% accuracy.
2. Pairwise tasks
In this work, we focus on “pairwise” tasks, meaning that the input space decomposes as X =
X 1 × X 2 .
For instance, X 1 could be questions and
X 2 could be paragraphs for open-domain question answering, and X
= X 2 could be questions for question deduplication.
We create the unlabeled all-pairs training set D trainall by taking the cross product of a subset from X 1 and a subset from X 2 .
We follow the same procedure to form the all-pairs test set, D testall .
As is standard practice, we ensure that the train and test all-pairs sets are disjoint.
Many pairwise tasks require high average precision on all-pairs test data.
A question deduplication system must compare a new question with all previously asked questions to determine if a duplicate exists.
An open-domain question-answering system must search through all available documents for one that answers the question.
In both cases, the all-pairs distribution is extremely imbalanced, as the vast majority of pairs are negatives, while standard datasets are artiﬁcially balanced.
Results training on heuristic datasets
In this section, we show that state-of-the-art models trained on two standard pairwise classiﬁcation datasets—QQP and WikiQA—do not generalize well to our extremely imbalanced all-pairs test data, which we create from an original dataset by imputing a negative label for all pairs that are not marked as positive.
Both QQP and WikiQA were collected using static heuristics that attempt to ﬁnd points x ∈ X that are more likely to be positive.
These heuristics are necessary because uniformly sampling from X is impractical due to the label imbalance: if the proportion of positives is 10 − 4 , then random sampling would have to label 10, examples on average to ﬁnd one positive example.
Standard models can achieve high test accuracy on test data collected with these heuristics, but fare poorly when evaluated on all-pairs data derived from the same data source (Section 3.2).
Manual inspection conﬁrms that these models often make surprising false positive errors (Section 3.3).
3. Experimental setup 3.1. Evaluation
We evaluate models on both heuristically balanced test data and our imbalanced all-pairs test data.
Heuristically balanced evaluation.
Let D pos denote the set of all positive examples, and D statedneg denote the set of stated negative examples —negative examples in the original heuristically collected dataset.
We deﬁne the stated test dataset D test heur as the pairs in the original balanced dataset that are also in our deﬁned test dataset: ( D pos ∪ D statedneg ) ∩ D testall .
This is similar to the original QQP test data, but with a different train/test split.
We use task-speciﬁc evaluation metrics described in the next section.
All-pairs evaluation.
All-pairs evaluation metrics depend on the label of every pair in D testall .
We approximate these labels by imputing (possibly noisy) labels on all pairs using the available labeled data, as described in the next section.
In
Section 3.3, we manually label examples to conﬁrm our results from this automatic evaluation.
Computing the number of false positives
FP ( S, γ ) requires enumerating all negative examples in D testall , which is too computationally expensive with our datasets.
To get an unbiased estimate of FP ( S, γ ) , we could randomly subsample D testall , but the resulting estimator has high variance.
We instead compute an unbiased estimator that uses importance sampling.
In particular, we combine counts of errors on a set of “nearby negative
” examples D testnear ⊆
D testall , pairs of similar utterances on which we expect more false positives to occur, and random negatives D testrand sampled uniformly from negatives in D testall \ D testnear .
Details are provided in
Appendix A.2. 3.1. Datasets
Quora Question Pairs (QQP).
The task for
QQP (Iyer et al., 2017) is to determine whether two questions are paraphrases.
The non-paraphrase pairs in the dataset were chosen heuristically, e.g., by ﬁnding questions on similar topics.
We impute labels on all question pairs by assuming that two questions are paraphrases if and only if they are equivalent under the transitive closure of the equivalence relation deﬁned by the labeled paraphrase pairs.
2 We randomly partition all unique questions into train, dev, and test splits, ensuring that no two questions that were paired (either in positive or negative examples) in the original dataset end up in different splits.
Since every question is a paraphrase of itself, we deﬁne D trainall as the set of distinct pairs of questions from the training questions, and deﬁne D devall and D testall analogously.
For heuristically balanced evaluation, we report accuracy and F1 score Split Positives Totalpairs
Ratio StatedNeg. NearbyNeg. QQP Train 124, 38B 1:300K 132, - Dev 60, 8.8B 1:150K 61, 13.1M Test 43, 8.5B 1:190K 60, 12.8M
WikiQA Train 1, 56M 1:53K 19, - Dev 7.8M 1:56K 2, 29,511 Test 17M 1:57K 5, 63, on D testheur , as in Wang et al.
WikiQA. The task for WikiQA (Yang et al., 2015) is to determine whether a question is answered by a given sentence.
The dataset only includes examples that pair a question with sentences from a Wikipedia article believed to be relevant based on click logs.
We impute labels by assuming that question-sentence pairs not labeled in the dataset are negative.
We partition the questions into train, dev, and test, following the original questionbased split of the dataset, and then take the direct product with the set of all sentences in the original dataset to form the train, dev, and test sets.
For all
WikiQA models, we prepend the title of the source article to the sentence to give the model information about the sentence’s origin, as in Lee et al.
For heuristically balanced evaluation, we report two evaluation metrics.
Following standard practice, we report clean mean average precision (c-MAP), deﬁned as MAP over “clean” test questions—questions that are involved in both a positive and negative example in D testheur (Garg et al., 2020).
We also report F1 score across all examples in D testheur (a-F1), which unlike c-MAP considers the more realistic setting where questions may not be answerable given the available article.
This introduces more label imbalance, as positives make up 6% of D testheur but 12% of clean examples.
The original WikiQA paper advocated a-F
(Yang et al., 2015), but most subsequent papers do not report it (Shen et al., 2017;
Yoon et al., 2019; Garg et al., 2020).
Data statistics.
Table 1 shows dataset statistics.
Models in this section are trained on the stated training dataset
[Equation]
the set of all positives and heuristic negatives in the train split.
For all-pairs evaluation, both QQP and
WikiQA have extreme label imbalance: Positive [Table] examples make up between 1 in 50,000 (WikiQA) and 1 in 200,000 (QQP) of the test examples.
3.1. Models
We train four state-of-the-art models that use BERTbase (Devlin et al., 2019), XLNet-base (Yang et al., 2019), RoBERTa-base (Liu et al., 2019), and ALBERT-base-v2 (Lan et al., 2020), respectively.
As is standard, all models receive as input the concatenation of x 1 and x 2 separated by a special token; we refer to these as concatenationbased (C ONCAT ) models.
We train on binary crossentropy loss for 2 epochs on QQP and 3 epochs on WikiQA, chosen to maximize dev all-pairs AP for RoBERTa.
We report the average over three random seeds for training.
3. Evaluation results
As shown in Table 2, state-of-the-art models trained only on stated training data do well on heuristically balanced test data but poorly on the extremely imbalanced all-pairs test data.
On QQP, the best model gets 80 . 2% F1 on heuristically balanced test examples.
However, on all-pairs test data, the best model can only reach 3 .
5% precision at a modest 20% recall.
On WikiQA, our best c-MAP of 84 .
6% is higher than the best previously reported c-MAP without using additional question-answering data, 83 .
6% (Garg et al., 2020).
However, on all-pairs test data, the best model gets 6 . 5% precision at 20% recall.
All-questions F1 on heuristically balanced data is also quite low, with the best model only achieving 53 .
Since a-F1 evaluates on a QQP, C ONCAT BERT trained on D trainheur [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
WikiQA, C ONCAT BERT trained on D trainheur
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure] more imbalanced distribution than c-MAP, this further demonstrates that state-of-the-art models deal poorly with test-time label imbalance.
Compared with a-F1, all-pairs evaluation additionally shows that models make many mistakes when evaluated on questions paired with less related sentences; these examples should be easier to identify as negatives, but are missing from D trainheur .
3. Manual veriﬁcation of imputed negatives
Our all-pairs evaluation results are based on automatically imputed negative labels, rather than the gold label evaluation metrics.
To check the validity of our results, we manually labeled putative false positive errors—examples that our model labeled positively but for which the imputed label was negative—to more accurately estimate precision.
We focused on the best QQP model and random seed combination on the development set, which got 8 . 2% precision at 20% recall.
For this recall threshold, we manually labeled 50 randomly chosen putative false positives from D devnear , andmore from D devrand .
In 72% and 92% of cases, respectively, the imputed label was correct and the model was wrong.
Extrapolating from these results, we estimate the true precision of the model to be 9 .
5% , still close to our original estimate of 8 . 2% .
See Appendix A.3 for more details.
For the remainder of the paper, we simply use the imputed labels, keeping in mind this may underestimate precision.
Figure 2 shows real false positive predictions at 20% recall for the best QQP and WikiQA models.
For QQP, models often make surprising errors on pairs of unrelated questions (ﬁrst two examples), as well as questions that are somewhat related but distinct (third example).
For WikiQA, models often predict a positive label when something in the sentence has the same type as the answer to the question, even if the sentence and question are unrelated.
While these pairs seem easy to classify, the heuristically collected training data lacks coverage of these pairs, leading to poor generalization.
Active learning for pairwise tasks
As shown above, training on heuristically collected balanced data leads to low average precision on all pairs.
How can we collect training data that leads to high average precision?
We turn to active learning, in which new data is chosen adaptively based on previously collected data.
Adaptivity allows us to ignore the vast majority of obvious negatives (unlike random sampling) and iteratively correct the errors of our model (unlike static data collection) by collecting more data around the model’s decision boundary.
4. Active learning
Formally, an active learning method takes in an unlabeled dataset D trainall ⊆
X .
Data is collected in a series of k > 1 rounds.
For the i th round, we choose a batch B i ⊆
D trainall of size n i and observe the outcome as the labels { ( x, y ( x )) : x ∈ B i } .
The budget n is the total number of points labeled, i.e., n = (cid:80) ki
=1 n i .
This process is adaptive because we can choose batch B i based on the labels of the previous i − 1 batches.
Static data collection corresponds to setting k = 1 .
Uncertainty sampling
The main active learning algorithm we use is uncertainty sampling (Lewis and Gale, 1994), which is simple, effective, and commonly used in practice (Settles, 2009).
Uncertainty sampling ﬁrst uses a static data collection method to select the seed set B 1 .
For the next k −rounds, uncertainty sampling trains a model on all collected data and chooses B i to be the n i unlabeled points in D train all on which the model is most uncertain.
For binary classiﬁcation, the most uncertain points are the points where p θ ( y = 1 | x ) is closest to 12 .
Note that a brute force approach to ﬁnding B i requires evaluating p θ on every example in D trainall , which can be prohibitively expensive.
In balanced settings, it sufﬁces to choose the most uncertain point from a small random subset of D trainall (Ertekin et al., 2007); however, this strategy works poorly in extremely imbalanced settings, as a small random subset of D trainall is unlikely to contain any uncertain points.
In Section 4.2, we address this computational challenge with a bespoke model architecture.
Adaptive retrieval.
We also use a related algorithm we call adaptive retrieval, which is like uncertainty sampling but queries the n i unlabeled points in D trainall with highest p θ ( y = 1 | x ) (i.e., pairs the model is most conﬁdent are positive).
Adaptive retrieval can be seen as greedily maximizing the number of positive examples collected.
4. Modeling and implementation
We now fully specify our approach by describing our model, how we ﬁnd pairs in the unlabeled pool
D trainall to query, and how we choose the seed set B 1 .
In particular, a key technical challenge is that the set of training pairs D trainall is too large to enumerate, as it grows quadratically.
We therefore require an efﬁcient way to locate the uncertain points in
D trainall .
We solve this problem with a model architecture C OSINE BERT that enables efﬁcient nearest neighbor search (Gillick et al., 2019).
4.2. Model
Given input x = ( x 1 , x 2 ) , C OSINE BERT embeds x 1 and x 2 independently and predicts p θ ( y = 1 | x ) based on vector-space similarity.
More precisely, p θ ( y = 1 | x ) = σ (cid:18) w · e θ ( x 1 ) · e θ ( x 2 ) (cid:107) e θ ( x 1 ) (cid:107)(cid:107) e θ ( x 2 ) (cid:107) + b (cid:19) , (7) where σ is the sigmoid function, w > 0 and b are learnable parameters, and e θ : X 1 ∪ X 2 → R d is a learnable embedding function.
In other words, we compute the cosine similarity of the embeddings of x 1 and x 2 , and predict y using a logistic regression model with cosine similarity as its only feature.
We deﬁne e θ as the ﬁnal layer output of a BERT model (Devlin et al., 2019) mean-pooled across all tokens (Reimers and Gurevych, 2019).
5 Gillick et al.
(2019) used a similar model for entity linking.
4.2. Finding points to query
Next, we show how to choose the batch B i of points to query, given a model p θ ( y | x ) trained on data from batches B 1 , . . .
B i − 1 .
Recall that uncertainty sampling chooses the points x for which for which p θ ( y = 1 | x ) is closest to 12 , and adaptive retrieval chooses the points x with largest p θ ( y = 1 | x ) .
Since the set of positives is very small compared to the size of D trainall , the set of uncertain points can be found by ﬁnding points with the largest p θ ( y = 1 | x ) , thus ﬁltering out the conﬁdent negatives, and then selecting the most uncertain from those.
To ﬁnd points with largest p θ ( y = 1 | x ) , we leverage the structure of our model.
Since w > 0 , p θ ( y = 1 | x ) is increasing in the cosine similarity of e θ ( x 1 ) and e θ ( x 2 ) .
Therefore, it sufﬁces to ﬁnd pairs ( x 1 , x 2 ) that are nearest neighbors in the embedding space deﬁned by e θ .
In particular, for each x 1 ∈ X 1 , we use the Faiss library (Johnson et al., 2017) to retrieve a set N ( x 1 ) containing the m nearest neighbors in X 2 , and deﬁne D trainclose to be the set of all pairs ( x 1 , x 2 ) such that x 2 ∈ N ( x 1 ) .
We then iterate through D trainclose to ﬁnd either the most uncertain points (for uncertainty sampling) or points with highest cosine similarity (for adaptive retrieval).
Note that this method only requires embedding the number of distinct elements that appear in the training set, rather than the total number of pairs, the requirement for jointly embedding all pairs.
4.2. Choosing the seed set
Recall that both of our active learning techniques require a somewhat representative initial seed set
B 1 to start the process. We use the pre-trained
BERT model as the embedding e θ and select the n 1 pairs with largest p θ ( y = 1 | x ) .
Recall that w > 0 , so this amounts to choosing the pairs with highest cosine similarity.
Active learning experiments 5. Experimental details
We simulate active learning with the imputed labels so that we can compare different algorithms without performing expensive gold label collection for each algorithm.
We collect n 1 = 2048 examples in the seed set, and use k = 10 rounds of [Table] [Table] active learning for QQP and k = 4 for WikiQA, as WikiQA is much smaller.
At round i , we query n i = n 1 · (3 / 2) i − 1 new labels.
The exponentially growing n i helps us avoid wasting queries in early rounds, when the model is worse
[Table]
and also makes training faster in the early rounds.
These choices imply a total labeling budget n of 232,100 for QQP and 16,640 for WikiQA.
For both datasets, n is slightly less than | D trainheur | (257,421 for QQP and 20,360 for WikiQA), thus ensuring a meaningful comparison with training on heuristic data.
We retrieve m = 1000 nearest neighbors per x 1 ∈ X 1 for QQP and m = 100 for WikiQA.
We run all active learning experiments with three different random seeds and report the mean.
Training details are given in Appendix A.
5. Main results
We now compare the two active learning methods, adaptive retrieval and uncertainty sampling, with training on D trainheur and two other baselines.
Random sampling queries n pairs uniformly at random, which creates a very imbalanced dataset.
Static retrieval queries the n most similar pairs using the pre-trained BERT embedding, similar to Section 4.2.3.
Table 3 shows all-pairs evaluation for C OSINE BERT trained on these datasets.
The two active learning methods greatly outperform other methods: Uncertainty sampling gets 32 . 5% AP on QQP and 20 .
1% on WikiQA, while the best static data collection method, static retrieval, gets only 25 . 1% AP on QQP and 8 . 2% AP on
WikiQA.
Recall from Table 2 that C ONCAT BERT only achieved 2 .
4% AP on both QQP and WikiQA.
When trained on the same data as C ONCAT BERT,
C OSINE BERT achieves much higher AP on QQP ( 15 . 4% ) but slightly lower AP on WikiQA ( 2 . 2% ).
Uncertainty sampling slightly outperforms adaptive retrieval on both datasets.
Achieving high precision across all pairs requires collecting both enough positive examples and useful negative examples.
Compared to random sampling and static retrieval, active learning collects many more positive examples, as shown in
Table 4.
D trainheur contains all positive examples, but models trained on it still have low AP on all pairs.
We conclude that the negative examples in D trainheur are insufﬁcient for generalization to all pairs, while active learning chooses more useful negatives.
Manual veriﬁcation of imputed negatives
As in Section 3.3, we manually labeled putative
QQP false positives at the threshold where recall is 20% for C OSINE BERT trained on either stated data or uncertainty sampling data.
For each, we labeled 50 putative false positives from D devnear , and all putative false positives from D devrand (12 for stated data, 0 for uncertainty sampling).
C OSINE BERT trained on D trainheur .
67% (8 of 12) of the putative false positives on D devrand were actual errors by the model, but only 36% of putative false positives on D devnear were errors.
Extrapolating from these results, we update our estimate of development set precision at 20% recall from 28 .
4% to 41 .
Overall, this model makes some more reasonable mistakes than the C ONCAT BERT model, though its precision is still not that high.
C OSINE BERT model with uncertainty sampling.
Only 32% of putative false positives from
D devnear were real errors, signiﬁcantly less than the 72% for C ONCAT BERT trained on D trainheur ( p = 7 × 10 − 5 , Mann-Whitney U test).
Extrapolating from these results, we update our estimate of development set precision at 20% recall from 55 .
1% to 79 .
3% , showing that uncertainty sampling yields a more precise model than our imputed labels indicate.
In fact, this model provides a high-precision way to identify paraphrase pairs not annotated in the original dataset.
[Table]
5. Comparison with stratiﬁed sampling
Next, we further conﬁrm that having all the positive examples is not sufﬁcient for high precision.
In Table 5, we compare with two variants of stratiﬁed sampling, in which positive and negative examples are independently subsampled at a desired ratio (Attenberg and Provost, 2010).
First, we randomly sample positive and negative training examples to match the number of positives and negatives collected by uncertainty sampling, the best active learning method for both datasets (“Strat. match” in Table 5).
Second, we trained on all positive examples and added negatives to match the number of positives on QQP or match the active learning total budget on WikiQA (“Strat.
all pos.”).
For
QQP, this yielded a slightly larger dataset than the ﬁrst setting.
Note that stratiﬁed sampling requires oracle information: It assumes the ability to sample uniformly from all positives, even though this set is not known before data collection begins.
Nonetheless, stratiﬁed sampling trails uncertainty sampling by more than 10 AP points on both datasets.
Since stratiﬁed sampling has access to all positives, active learning must be choosing more informative negative examples.
5. Training other models on collected data
For QQP, data collected with active learning and
C OSINE BERT is useful for training other models on the same task.
Table 6 shows that C ONCAT BERT does better on data collected by active learning—using C OSINE BERT—compared to the original dataset or static retrieval.
C ONCAT BERT performs best with stratiﬁed sampling; recall that this is not a comparable data collection strategy in our setting, as it requires oracle knowledge. C OSINE BERT outperforms C ONCAT BERT in all training conditions; we hypothesize that the cosine similarity structure helps it generalize more [Figure] AP
[Figure]
AP Stated [Figure] [Table]
Static Ret.
[Table] Stratiﬁed [Table] Uncertainty
[Table] [Figure] robustly to pairs of unrelated questions. However,
C OSINE BERT trained on stated data does not do as well on WikiQA, as shown in Table 3.
5. Data efﬁciency
Adaptivity is crucial for getting high AP with less labeled data.
Static retrieval, the best static data collection method, gets 21 .
9% dev AP on QQP with the full budget of 232,100 examples.
Uncertainty sampling achieves a higher dev AP of 22 . 6% after collecting only 16,640 examples, for a 14 × data efﬁciency improvement.
See Appendix B. for further analysis.
5. Effect of seed set
Our method is robust to choice of the initial seed set for uncertainty sampling.
We consider using stated data as the seed set, instead of data chosen via static retrieval.
As shown in Figure 3, seeding with stated data performs about as well as static retrieval in terms of AP.
Since the stated data artiﬁcially overrepresents positive examples, the model trained on stated data is initially miscalibrated—the points it is uncertain about are actually almost all negative points. Therefore, uncertainty sampling initially collects very few additional positive examples. Over time, adaptively querying new data helps correct for this bias.  Discussion and related work
In this paper, we have studied how to collect training data that enables generalization to extremely imbalanced test data in pairwise tasks.
State-ofthe-art models trained on standard, heuristically collected datasets have very low average precision when evaluated on imbalanced test data, while active learning leads to much better average precision.
Gillick et al.
(2019) propose a similar model for entity linking and mine hard negative examples, an approach related to adaptive retrieval.
However, they have abundant labeled data, whereas we study data collection with a limited labeling budget.
Work in information retrieval often attempts to maximize precision across all pairs of test objects.
Machine learning models are commonly used to re-rank candidate pairs from an upstream retriever (Chen et al., 2017; Nogueira and Cho, 2019), while our method learns embeddings to improve the initial retrieval step.
Distant supervision has been used to train end-to-end retrieval models for question answering (Lee et al., 2019), but does not extend to other tasks like paraphrase detection.
Other work on duplicate question detection on community QA forums trains on labels generated by forum users (dos Santos et al., 2015).
Hoogeveen et al.
(2016) show that these datasets tend to have many false negatives and suggests additional labeling to correct this problem; active learning provides one way to choose informative pairs to label.
Extreme label imbalance is an important challenge in many non-pairwise NLP tasks, including document classiﬁcation (Lewis et al., 2004) and relation extraction (Zhang et al., 2017).
Most prior work focuses on sampling a ﬁxed training dataset (Chawla et al., 2004; Sun et al., 2009; Dendamrongvit and Kubat, 2009), whereas our work explores data collection.
Attenberg and Provost (2010) ﬁnd stratiﬁed sampling outperforms active learning in non-pairwise imbalanced tasks, primarily due to the difﬁculty of ﬁnding a useful seed set.
We ﬁnd pre-trained embeddings effective for seed set collection in pairwise tasks.
Zhang et al.
(2019) found that the frequency of questions in QQP leaks information about the label.
Evaluating on all pairs avoids such artifacts, as every test utterance appears in the same number of examples.
Zhang et al.
re-weight the original dataset to avoid these biases, but re-weighting cannot compensate for the absence of some types of negative examples, unlike active learning.
Many pairwise datasets are generated by asking crowdworkers to generate part or all of the input x (Bowman et al., 2015; Mostafazadeh et al., 2016).
Having crowdworkers generate text increases the risk of introducing artifacts (Schwartz et al., 2017;
Poliak et al., 2018), while our pool-based approach considers the entire distribution of utterance pairs.
We use active learning, speciﬁcally uncertainty sampling (Lewis and Gale, 1994), to create a balanced training set that leads to models that generalize to the full imbalanced distribution.
Ertekin et al.
(2007) argues that active learning is capable of providing balanced classes to the learning algorithm by selecting examples close to the decision boundary.
Furthermore, active learning can generalize to the full distribution, both empirically (Settles, 2009;
Yang and Loog, 2018) and theoretically (Balcan et al., 2007; Balcan and Long, 2013; Mussmann and Liang, 2018).
Finally, this paper addresses two central concerns in NLP today: How to construct fair but challenging tests of generalization (Geiger et al., 2019), and how to collect training data in a way that improves generalization.
Evaluating on extremely imbalanced all-pairs data has several advantages over other tests of generalization.
Our examples are realistic and natural, unlike adversarial perturbations (Ebrahimi et al., 2018;
Alzantot et al., 2018), and diverse, unlike hand-crafted tests of speciﬁc phenomena (Glockner et al., 2018; Naik et al., 2018;
McCoy et al., 2019).
Since we allow querying the label of any training example, generalization to our test data is achievable, while out-of-domain generalization (Levy et al., 2017; Yogatama et al., 2019; Talmor and Berant, 2019) may be statistically impossible.
Our work thus offers a natural, challenging, and practically relevant testbed to study both generalization and data collection. Reproducibility.
Code and data needed to reproduce all results can be found on the CodaLab platform at https://bit.ly/2GzJAgM .
Acknowledgments
This work was supported by a PECASE Award,
NSF Award Grant no. 1805310, and NSF Graduate Fellowship DGE-1656518. We thank Chris Manning, Michael Xie, Dallas Card, and other members of the Stanford NLP Group for their helpful comments. A Experimental details A. Training details
At each round of active learning, we train forepochs.
We train without dropout, as dropout artiﬁcially lowers cosine similarities at training time.
We apply batch normalization (Ioffe and Szegedy, 2015) to the cosine similarity layer to rescale the cosine similarities, as they often are very close to 1.
We initialize w and b so that high cosine similarities correspond to the positive label, and constrain w to be nonnegative during training.
We use a maximum sequence length of 128 word piece tokens.
To compensate for BERT’s low learning rate, we increased the learning rate on the w and b parameters by a factor of 10 4 .
Below in Table 7, we show hyperparameters for training.
Hyperparameters were tuned on the development set of QQP; we found these same hyperparameters also worked well for WikiQA, and so we did not tune them separately for WikiQA.
In most cases, we used the default hyperparameters for BERT.
[Table]
At the end of training, we freeze the embeddings e θ and train the output layer parameters w and b to convergence, to improve uncertainty estimates for uncertainty sampling.
This process amounts to training a two-parameter logistic regression model.
We optimize this using (batch) gradient descent with learning rate 1 and 10 , 000 iterations.
When training this model, we normalize the cosine similarity feature to have zero mean and unit variance across the training dataset.
Training this was very fast compared to running the embedding model.
Each experiment was conducted with a single
GPU, most commonly a TITAN V or TITAN Xp.
Running one complete uncertainty sampling experiment (i.e., 10 rounds of data collection and model training for QQP, 4 for WikiQA) on a machine with one TITAN V GPU takes about 9 hours for
QQP and about 30 minutes for WikiQA.
Recall that
C OSINE BERT only adds two additional parameters, w and b , on top of a BERT model; we use the uncased BERT-base pre-trained model which has 110M parameters.
A. Evaluation details
To evaluate a given scoring function S at threshold γ on a test set D testall , we must compute the number of true positives TP ( S, γ ) , false positives FP ( S, γ ) , and false negatives FN ( S, γ ) .
True positives and false negatives are computationally easy to compute, as they only require evaluating S ( x ) on all the positive inputs x in D testall .
However, without any structural assumptions on S , it is computationally infeasible to exactly compute the number of false positives, as that would require evaluating S on every negative example in D testall , which is too large to enumerate.
Therefore, we devise an approach to compute an unbiased, low-variance estimate of FP ( S, γ ) .
Recall that this term is deﬁned as
[Equation]
where D testneg denotes the set of all negative examples in D testall .
One approach to estimating FP ( S, γ ) would be simply to randomly subsample D testneg to some smaller set R , count the number of false positives in R , and then multiply the count by | D testneg | / | R | .
This would be an unbiased estimate of D test neg , but has high variance when the rate of false positive errors is low.
For example, if | D testneg | = 10 10 , | R | = 10 6 , and the model makes a false positive error on 1 in 10 6 examples in D testneg , then FP ( S, γ ) =
However, with probability
[Equation]
R will contain no false positives, so we will estimate FP ( S, γ ) as 0 .
A similar calculation shows that the probability of having exactly one false positive in R is also roughly 1 /e , which means that with probability roughly 1 − 2 /e ≈ 0 . 264 , we will have at least two false positives in R , and therefore overestimate FP ( S, γ ) by at least a factor of two.
To get a lower variance estimate ˆ FP ( S, γ ) , we preferentially sample from likely false positives and use importance weighting to get an unbiased estimate of FP ( S, γ ) .
In particular, we construct
D testnear to be the pairs in D testall with nearby pretrained BERT embeddings, analogously to how we create the seed set in Section 4.2.3.
Points with nearby BERT embeddings are likely to look similar are therefore more likely to be false positives.
Note that
[Equation]
where we deﬁne w rand =
| D testneg | − | D testnear | .
We can compute the ﬁrst term exactly, since
D testnear is small enough to enumerate, and approximate the second term as
[Equation]
where D testrand is a uniformly random subset of D testneg \
D testnear .
Therefore, our ﬁnal estimate is
[Equation]
A. Incorporating manual labels
In Section 3.3, we manually label examples that were automatically labeled as false positives, and use this to improve our estimates of the true model precision.
We manually label randomly chosen putative false positives from both D devnear and D devrand , and use this to estimate the proportion of putative false positives in each set that are real false positives.
Let ˆ p near denote the estimated fraction of putative false positives in D devnear that are real false positives, and ˆ p rand be the analogous quantity for
D devrand .
Our updated estimate ˆ FP manual ( S, γ ) is then deﬁned as
[Equation]
We then compute precision using ˆ FP manual ( S, γ ) in place of ˆ FP ( S, γ ) .
A. Comparison with GLUE QQP data
For a few reasons, our QQP in-domain accuracy numbers are lower than those on the GLUE leaderboard, which has accuracies in the low 90 ’s. First, our training set is smaller (257K examples versus 364K).
Second, our split is more challenging because the model does not see the same questions or even same paraphrase clusters at training time and test time.
Finally, our test set is more balanced ( 58% negative) than the GLUE QQP dev set ( 63% negative; test set balance is unknown).
As a sanity check, we conﬁrmed that our RoBERTa implementation can achieve 91 .
5% dev accuracy when trained and tested on the GLUE train/dev split, in line with previously reported results (Liu et al., 2019).
B Additional experimental results B. Learning curves and data efﬁciency
[Figure]
In Figure 4a
[Figure]
we plot average precision on the
QQP dev set for our model after each round of uncertainty sampling.
For comparison, we show a model trained on the same amount of data collected via static retrieval, the best-performing static data collection method.
Uncertainty sampling leads to higher AP with much less data.
For example, uncertainty sampling only needs to collect 16, examples to surpass the average precision of static retrieval collecting all 232,100 examples, for a 14 × data efﬁciency improvement.
A big factor for the success of uncertainty sampling is its ability to collect many more positive examples than static retrieval, as shown in Figure 4b.
Static retrieval collects fewer positives over time, as it exhausts the set of positives that are easy to identify.
However, uncertainty sampling collects many more positives, especially after the ﬁrst round of training, because it improves its embeddings over time.