Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classification (2024)

Abstract: Few-shot learning benchmarks are critical for evaluating modern NLP techniques. It is possible, however, that benchmarks favor methods which easily make use of unlabeled text, because researchers can use unlabeled text from the test set to pretrain their models. Given the dearth of research on this potential problem, we run experiments to quantify the bias caused by pretraining on unlabeled test set text instead of on unlabeled, independently drawn text. Controlled few-shot and zero-shot experiments on 25 classification tasks and 3 language models—BERT, GPT-2, and Mistral 7B—do not find evidence of overoptimism. Furthermore, we demonstrate the importance of repeated subsampling when studying few-shot text classification, and recommend that few-shot learning benchmarks include multiple training folds. Code and data are available here: https://github.com/kddubey/pretrain-on-test .

## introduction it is common for nlp benchmarks to release text from the test set, as researchers can submit a file of predictions instead of submitting code. a potential concern is that researchers can use this text during training. consider the real-world annotated fewshot tasks (raft) benchmark @xcite , which contains "few-shot" text classification taskstasks where the training set contains a relatively small number of labeled examples. below is an excerpt from the raft paper (emphasis added): for each task, we release a public training set with 50 examples and a larger unlabeled test set. we encourage unsupervised pre-training on the unlabelled examples and open-domain information retrieval. in the raft competition, a model is evaluated by scoring its predictions on the same set of unla-beled text which the model may have been trained on (using an unsupervised training procedure). it is wrong to train a model on test set features with their labels and then evaluate on the test set when one needs to estimate performance on outof-sample data. test set performance would be overoptimistic @xcite . this fact is widely known. but what if, as encouraged by @xcite , a model is trained on test set features without test set labels? this paper studies this question for the domain of few-shot text classification. ## motivation nlp benchmarks for few-shot learning are prevalent, as having only a handful of labeled examples is more realistic. one consideration when designing these benchmarks is that some few-shot approaches can-at least theoretically-use unlabeled text from the test set. with pattern-exploiting training @xcite , for example, one can train the final classifier on test set text with soft labels predicted by an ensemble of supervised models. with pre-trained prompt tuning @xcite , one can pretrain the language model (lm) on unlabeled test set text before prompt-tuning on the labeled training set. a more classical approach would be to train a word2vec model @xcite on unlabeled test set text, run this model on training text to get embeddings, and finally train a classifier on these embeddings with labels from the training set. for other few-shot approaches, such as setfit @xcite and in-context learning with llms (as popularized by @xcite , it is more common to only use labeled text. while the ability to exploit unlabeled text is useful, applying this ability to test set text could be substantively different than applying it to text which is statistically independent of the test set. this difference in methodology may be more concerning in 1 the few-shot setting than in the many-shot setting. it is conceivable that differences between few-shot methods are due just as much to how unlabeled text is used as they are to how the few, labeled examples are used. this raises the question: does pretraining a model on a benchmark's unlabeled test set text inflate the model's performance on that benchmark? ## related work as indicated by the quote in §1, the raft benchmark implicitly assumes that the answer is no. the validity of using test set features is not a fringe opinion. the popular textbook by @xcite contains the following passage without a reference or evidence (emphasis added): there is one qualification: initial unsupervised screening steps can be done before samples are left out. for example, we could select the 1000 predictors with highest variance across all 50 samples, before starting cross-validation. since this filtering does not involve the class labels, it does not give the predictors an unfair advantage. the opposite opinion-that exploiting unlabeled test set features is unfair-may align more closely with best practices. for example, @xcite contains the following criticism of another study when comparing performances on a text classification task: @xcite report a higher number (97.42) on imdb, but they train their word vectors on the test set. @xcite argue that benchmarks which release unlabeled test set text can be compromised, but do not discuss potential problems with using unlabeled test set text by itself. moscovich and rosset (2022) contains experiments and theory for unsupervised methods which are common to tasks involving tabular data. they find that estimators of out-of-sample performance which were subject to these methods may be biased positively or negatively, depending on the parameters of the problem. they recommend further research on this bias in more domains, particularly when dealing with small sample sizes and highdimensional data. ## experimental design we study whether pretraining on unlabeled test set text biases test set performance for 25 diverse text classification tasks and two types of lms: bert @xcite and gpt-2 @xcite . appendix a describes each task. the goal of the experiment is to first establish that pretraining is beneficial, in line with @xcite . second, given that pretraining has a detectable effect, the experiment measures the accuracy difference between using test set text for the pretraining stage-an arguably unfair methodology-and using text which is independent of the test set-an inarguably fair methodology. in more detail, the experiment starts by subsampling three separate sets of data from the full sample of data for a given text classification task: next, three accuracy estimators are computed. procedures used to obtain them are described below. ## analysis reporting means is not enough, especially when studying few-shot learning. appendix d.2 demonstrates that there is considerable variance, despite pairing the accuracy estimators. 2 while these visualizations tell us about how raw accuracy differences vary, they do not tell us how the mean accuracy difference varies. we seek a neat answer to the core questions: on this benchmark of 25 classification tasks, how much does the overall accuracy differ between two modeling techniques, and how much does this difference vary? one way to communicate the variance is to estimate the standard error of the mean difference across classification tasks. but the standard error statistic can be difficult to interpret @xcite . furthermore, its computation is not completely trivial due to the data's hierarchical dependency structure: each triple, (acc extra , acc test , acc base ), is drawn from (train, test), which is itself drawn from the given classification dataset. ## discussion figure 3 demonstrates that the average pretraining boost is significant in every configuration of the experiment. this finding replicates that from @xcite . after averaging across settings for m, n, and the 2 lm types, only two of the 25 classification tasks had a pretraining boost less than 0, and both were greater than -1%. 4 task-adaptive pretraining had the intended effect. as shown in figure 3 , the evaluation bias bounces inconsistently and insignificantly around 0. after averaging, 12 of the 25 classification tasks had a positive evaluation bias, 13 had a negative evaluation bias, and all tasks had an average evaluation bias less than 1% in absolute value. to avoid excessive averaging, we lemon-picked tasks which reported a bias of at least +3% in any experiment configuration. all tasks matching this criterion were from experiments with bert, as bert had greater training variance. if there were a task-dependent evaluation bias, one could expect that the bias is consistent across m or n within a task, or there is a consistent pattern with how the bias changes with m or n across tasks. figure 4 does not clearly support either of these hypotheses. @xcite found that the evaluation bias caused by unsupervised methods for tabular data converges to 0 as n increases. this finding is not confirmed by this experiment. figure 3 shows that within @xmath0, distributions of the evaluation bias hover around 0 across n. figure 4 also does not support a relationship between n and the evaluation bias for lemonpicked tasks. but far more experiments varying n are needed to thoroughly assess this insensitivity. for each of the 25 classification tasks and their subsamples, pretraining for 2 epochs instead of 1 resulted in a lower pretraining loss. the final pretraining loss is 20% lower on average, and the pretraining boost is negative, which indicates overfitting, as intended. figure 5 demonstrates that, despite overtraining, the evaluation bias hovers around 0. all 50 p-values from the test in (13) are greater than 0.5. foot_5 overtraining on unlabeled test set text causes test set performance to degrade to the same degree that overfitting on unlabeled independent text does. ## zero-shot text classification prompting an llm is a popular choice for solving nlp problems. these prompts can be pretrained on. for example, gemma 2 @xcite is intentionally pretrained on prompts from the lm-sys benchmark @xcite . to study a more modern prompting approach, the experiment in §4 is repeated with two modifications. first, task-adaptive pretraining is done on prompts-unlabeled texts with instructions for solving the task. second, classification training is not performed; train is unused. the furtherpretrained llm is immediately prompted to do the task on test. more specifically, pretraining is performed by adding a qlora adapter layer @xcite , 2024) to every linear layer in mistral 7b (jiang et al., 2023). perhaps notably, instructions mention the set of possible answers-the class names. figure @xcite shows that, while pretraining on prompts improves accuracy, pretraining on test set prompts does not increase test set accuracy compared to pretraining on independently drawn prompts. 12 of the 25 tasks had a positive evaluation bias and 13 had a negative evaluation bias. all 25 p-values from (13) are greater than 0.5; there is no evidence of a task-level evaluation bias. a limitation of this experiment is that it does not account for contamination. if mistral 7b's pretraining data included labeled or unlabeled parts of the datasets used here, the pretraining boost and evaluation bias may be diluted. ## conclusion task-adaptive pretraining on unlabeled test set text-instead of on unlabeled independent textdid not result in a consistent or significant evaluation bias. this appears to be the case when pretraining helps, when it hurts, and when pretraining is done on texts with instructions. for benchmarks which release unlabeled text from the test set, this finding does not completely absolve llm evaluations from scrutiny. the reason is that the boost from pretraining on unlabeled text-which is often significant-could be viewed as a type of evaluation bias, depending on how llms generalize. more concretely, suppose there is a benchmark and two llms, a and b. a was not pretrained on the benchmark's unlabeled test set text, while b was. with the perspective that llm benchmarks supply scores which are correlates of performance on real-world tasks-instead of indicators of performance solely on the benchmark's tasks-then b scoring higher on the benchmark than a may be a misleading signal. if pretraining on the benchmark's unlabeled text causes b to generalize better only within the distribution of the benchmark, then b's edge on this benchmark does not signal an edge in real-world tasks. knowing whether an llm was pretrained on unlabeled test set text is still important. one recommendation for designing few-shot benchmarks, which expands on the principle about robustness from @xcite , is based on the meta-analysis in §10: empirical studies of fewshot learning should consider including multiple, independent subsamples of training data. while a single training set combined with a large test set is sufficient for precise, unbiased estimation of out-of-sample performance, this estimator is conditional on the training set. in few-shot learning, the training set is, by definition, minimal. the estimator hides two sources of variance-that from the randomly drawn training set, and that from randomness inherent in the training procedure. figure 7 shows that this variance is large-enough to turn a methodology into a coin flip for two different training procedures. in-context learning with llms is also sensitive to the selection of few-shot examples @xcite . benchmarks which require training on multiple, independent subsamples would expose training variance.