O n the I mportance of A daptive D ata C ollection for E xtremely I mbalanced P airwise T asks (2020)

Abstract: Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., 99.99% of examples are negatives). In contrast, many recent datasets heuristically choose examples to ensure label balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only 2.4% average precision when evaluated on realistically imbalanced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently retrieve uncertain points from a very large pool of unlabeled utterance pairs. By creating balanced training data with more informative negative examples, active learning greatly improves average precision to 32.5% on QQP and 20.1% on WikiQA.

## introduction for most pairwise classification tasks in nlp, the most realistic data distribution has extreme label imbalance (e.g., 99.99% of examples have the same label). in question deduplication @xcite , the vast majority of pairs of questions from an online forum are not duplicates. in open-domain question answering @xcite , almost any randomly sampled document will not answer a given question. random pairs of sentences from a diverse distribution will have no relation between them in natural language inference @xcite , as opposed to an entailment or contradiction relationship. while past work has recognized the importance of label imbalance in nlp @xcite , many recently released datasets are heuristically collected to ensure label balance, generally for ease of training. for instance, * authors contributed equally. training evaluation heuristic, balanced dataset query access to imbalanced all-pairs dataset imbalanced all-pairs dataset adaptive framework static framework heuristic, balanced dataset sota models yield poor performance figure 1 : modern benchmarks often use heuristically balanced data for training and evaluation. we find that models trained on this data perform poorly on the very imbalanced all-pairs distribution and develop adaptive methods to collect training data for this setting. the quora question pairs (qqp) dataset @xcite was generated by mining non-duplicate questions that were heuristically determined to be nearduplicates. the snli dataset had crowdworkers generate inputs to match a specified label distribution @xcite . in this work, we show that models trained on heuristically balanced datasets deal poorly with natural label imbalance at test time: they have very low average precision on realistically imbalanced test data created by taking all pairs of test utterances. instead of heuristically producing static training datasets, we study adaptive data collection methods. in particular, we apply the pool-based active learning framework @xcite to extremely imbalanced pairwise tasks. at training time, the system has query access (i.e., the ability to collect a limited subset of the labels) to a large unlabeled dataset formed by taking all utterance pairs from a set of training utterances. for example, in the question deduplication setting, we might have a budget to annotate pairs of questions as "duplicate" or "not duplicate," and wish to train a model on this data to find new duplicate pairs with high precision. data collection for extremely imbalanced pairwise tasks is challenging because the pool of unlabeled examples is very large (as it grows quadratically in the number of utterances) and very few of the examples are positive. we collect balanced training data using uncertainty sampling, an adaptive method that queries labels for examples on which a model trained on previously queried data has high uncertainty @xcite . to lower the computational cost of searching for uncertain points, we propose combining active learning with a bert embedding model for which uncertain points can be located efficiently using nearest neighbor search. in this work, we empirically show that our use of adaptive data collection yields significant gains over static heuristics. in order to compare methods without collecting data separately for each method and each run, we perform retrospective data collection with imputed labels to simulate data collection. uncertainty sampling with our bert embedding model achieves 32.5% and 20.1% average precision for qqp and wikiqa, respectively. in contrast, state-of-the-art models trained on the original heuristically collected data each have only average precision of 2.4%. ## setting the pairwise tasks described above fall under a more general category of binary classification tasks, those with an input space x and output space {0, 1}. we assume the label y is a deterministic function of x, which we write y(x). a classification model p θ yields probability estimates p θ (y | x) where @xmath0. our setting has two aspects: the way training data is collected via label queries (section 2.1) and the way we evaluate the model p θ (y | x) by measuring average precision (section 2.2). this work focuses on pairwise tasks (section 2.3), which enables efficient active learning (section 4). ## results training on heuristic datasets in this section, we show that state-of-the-art models trained on two standard pairwise classification datasets-qqp and wikiqa-do not generalize well to our extremely imbalanced all-pairs test data, which we create from an original dataset by imputing a negative label for all pairs that are not marked as positive. both qqp and wikiqa were collected using static heuristics that attempt to find points @xmath1. these heuristics are necessary because uniformly sampling from x is impractical due to the label imbalance: if the proportion of positives is 10 -4 , then random sampling would have to label 10,000 examples on average to find one positive example. standard models can achieve high test accuracy on test data collected with these heuristics, but fare poorly when evaluated on all-pairs data derived from the same data source (section 3.2). manual inspection confirms that these models often make surprising false positive errors (section 3.3). ## active learning for pairwise tasks as shown above, training on heuristically collected balanced data leads to low average precision on all pairs. how can we collect training data that leads to high average precision? we turn to active learning, in which new data is chosen adaptively based on previously collected data. adaptivity allows us to ignore the vast majority of obvious negatives (unlike random sampling) and iteratively correct the errors of our model (unlike static data collection) by collecting more data around the model's decision boundary. ## active learning experiments 5.1 experimental details we simulate active learning with the imputed labels so that we can compare different algorithms without performing expensive gold label collection for each algorithm. we collect n 1 = 2048 examples in the seed set, and use @xmath2. for asymmetric tasks like nli where @xmath3, we would need to use separate encoders for the x1 and x2. the exponentially growing n i helps us avoid wasting queries in early rounds, when the model is worse, and also makes training faster in the early rounds. these choices imply a total labeling budget n of 232,100 for qqp and 16,640 for wikiqa. for both datasets, n is slightly less than |d train heur | (257,421 for qqp and 20,360 for wikiqa), thus ensuring a meaningful comparison with training on heuristic data. we retrieve @xmath4. we run all active learning experiments with three different random seeds and report the mean. training details are given in appendix a.