A Little Bit Is Worse Than None: Ranking with Limited Training Data (2020)

Abstract: Researchers have proposed simple yet effective techniques for the retrieval problem based on using BERT as a relevance classifier to rerank initial candidates from keyword search. In this work, we tackle the challenge of fine-tuning these models for specific domains in a data and computationally efficient manner. Typically, researchers fine-tune models using corpus-specific labeled data from sources such as TREC. We first answer the question: How much data of this type do we need? Recognizing that the most computationally efficient training is no training, we explore zero-shot ranking using BERT models that have already been fine-tuned with the large MS MARCO passage retrieval dataset. We arrive at the surprising and novel finding that “some” labeled in-domain data can be worse than none at all.

## introduction given a corpus c comprised of an arbitrary number of texts, the goal of the retrieval task is to generate a ranked list of k results for a user query q that maximizes some metric of interest. texts can differ in length: if the corpus is comprised of paragraphlength segments, the task is referred to as passage retrieval. otherwise, information retrieval (ir) researchers use the term document retrieval. bert @xcite has been successfully applied to the passage retrieval task by using it as a relevance classifier that reranks an initial list of candidate results @xcite , which are retrieved using bag-of-words queries and efficient exact-match scoring techniques such as bm25. as passages are usually shorter than the 512 token input length limit of bert, this solution is straightforward. even in cases where the candidate text exceeds this length limitation, @xcite showed that simply taking the bestscoring passage from a longer document as a proxy for the document score is an effective technique. in keeping with the theme of this workshop, these are simple yet effective approaches to tackling the retrieval problem. building on these two previous innovations, our work tackles the problem of training ranking models for specific domains (corpora) in a data and computationally efficient manner. while the most straightforward solution would be to gather relevance judgments on the target corpus, this is a nontrivial task. such ir test collections are usually produced via efforts like the text retrieval conferences (trecs) organized by the u.s. national institute for standards and technology. these collections are the result of community-wide efforts and beyond the capabilities of individual research teams. we consider the question of how much training data are needed to fine-tune an effective ranking model. does adapting a bert-based ranker to a new domain require trec-like levels of effort? the most data and computationally efficient training procedure is, of course, no training at allthat is, zero-shot learning. given the appeal of skipping the fine-tuning process altogether, we also explore how a zero-shot approach compares to finetuning on the target domain. there exists publicly available bert models that have already been finetuned with existing labeled data, for example, in the "model zoo" of huggingface's transformer library @xcite . we explore using these directly on our target corpora, and arrive at the interesting finding that a bit of labeled in-domain data can be worse than having none at all. in other words, if we don't have sufficient in-domain training data, it's better to simply adopt a zero-shot ranking approach using an already fine-tuned model: for this task, "few shot" is worse than "zero shot"! the primary contribution of this paper is an explication of this surprising finding that, to our knowledge, has not been reported in the literature. ## related work while bert's pretraining has reduced the burden of applying the model to downstream tasks, bert's maximum input length of 512 tokens presents a challenge for document retrieval. this length limitation prevents the straightforward application of bert to documents in typical corpora used for retrieval tasks, which are frequently longer. the obvious solution is to split documents into smaller passages, but this immediately raises the question of how to construct "passage-level" relevance labels from document-level labels. @xcite proposed the simple strategy of giving all passages the same label as the document (at training time) and aggregating passage scores (at inference time). their most effective approach, bert-maxp, uses the maximum passage score as the document score at inference (ranking) time. alternatively, this obstacle can be entirely avoided with a zero-shot approach: the model is fine-tuned on a passage retrieval dataset and then directly applied to the target corpus @xcite . for example, @xcite found that when bert was fine-tuned on a combination of (out-of-domain) datasets, the model exhibited state-of-the-art effectiveness (at the time) on robust04. @xcite confirmed this finding and further improved zero-shot effectiveness on robust04 by fine-tuning t5 @xcite on the ms marco passage dataset @xcite . @xcite investigated the use of adversarial regularization to prevent pre-bert neural models from learning representations closely tied to a specific domain. they found that training on a dataset fused from multiple domains was effective and can be further improved using adversarial regularization. in a supervised setting with transformers, finetuning on a related "intermediate" dataset before fine-tuning on the target dataset can be beneficial. @xcite was the first to show this for natural language inference tasks, and the evidence is consistent for retrieval tasks. @xcite showed that fine-tuning bert on bing search logs before fine-tuning on a trec dataset improved bert-maxp's effectiveness. similarly, @xcite found that bert-maxp also benefits from intermediate fine-tuning on ms marco. as expected, we encounter diminishing returns in effectiveness improvements as the amount of labeled training data increases; that is, increasing amounts of data are needed to obtain further improvements. @xcite demonstrated this on the ms marco passage ranking dataset. however, to the best of our knowledge, no previous work has investigated the effect of training data size for traditional trec-style datasets on bert-based models, which are smaller than ms marco by orders of magnitude. beyond ranking tasks, methods for tackling the limited labeled data issue using transfer learning have also been investigated. @xcite conducted supervised learning on the source dataset and unsupervised learning on the target dataset. ma et al. (2019) employed adversarial learning to generate pseudo-labels for target datasets. interestingly, both papers reported that directly transferring knowledge learned from a supervised out-of-domain dataset or unsupervised in-domain dataset to a target domain consistently underperforms supervised in-domain training without the intervention of special techniques (e.g., adversarial regularization). ## methodology in order to analyze the impact of fine-tuning a bert ranking model with limited training data, we sample standard benchmark datasets to simulate having less data available. rather than proposing a new model, we use the bert-maxp model @xcite due to its simplicity and demonstrated effectiveness on several datasets. to simulate the impact of having limited data, we prepare six different datasets that comprise relevance judgments sampled from the full dataset at a sampling rate @xmath0.1, 0.3, 0.5, 0.7, 0.9, 1.0}. the setting @xmath1.0 is equivalent to using the full dataset. specifically, given a dataset with n queries and m relevance judgements, the rsampled dataset contains roughly r×n queries and exactly r × m judgements. that is, queries (along with all their associated judgments) are dropped with a higher priority. this is accomplished by randomly dropping a query until doing so would result in fewer than r × m judgments. when this condition is reached, we loop over the remaining queries, randomly removing one judgment per query until there are exactly r × m judgments remaining. when we split our datasets into training, validation, and test folds for experiments, sampling is applied to only training and validation; we always calculate evaluation metrics using all available judgments. 4 experiments ## results and discussion in this section, we investigate bert-maxp's effectiveness on the robust04 and gov2 datasets when trained with limited data. figure 1 shows the model's effectiveness as the amount of training data increases both with and without first fine-tuning on the intermediate ms marco dataset. we also include bm25 (our first-stage retrieval), the unsupervised bm25+rm3 query expansion approach, and a zero-shot model in which bert-maxp is fine-tuned only on ms marco. these correspond to horizontal dotted lines in the plots. corresponding numerical values can be found in impact of dataset size. as expected, we observe in figure 1 that effectiveness generally increases as the amount of training data grows (i.e., as the sampling rate increases) on both collections regardless of whether the model is first fine-tuned on ms marco. the increase in effectiveness is most obvious when the amount of data is small, e.g., from @xmath2.1 to 0.3. effectiveness appears to plateau at @xmath3.7, which is surprising given that both corpora contain relatively small numbers of examples when compared to the sizes of datasets typically used in deep learning today. effectiveness even drops slightly when the full gov2 dataset is used (when not using ms marco). this suggests that the amount of relevance judgments available in both corpora is sufficient for fine-tuning the bert-maxp model. zero-shot effectiveness. surprisingly, fine-tuning bert with in-domain data is sometimes worse than zero shot, i.e., worse than not using in-domain data. from figure 1 , we see that this occurs up to @xmath4.3 on both robust04 and gov2 when no intermediate dataset is used. on robust04, this also occurs at @xmath5.1 even when the model is initialized with the ms marco checkpoint. at a sampling rate of @xmath6.3, fine-tuning on in-domain data directly (without ms marco) barely beats the traditional bm25+rm3 approach, which does not involve any neural network. when using intermediate data, however, the models are able to beat this non-neural baseline more easily. note that zero-shot bert is more effective than bm25+rm3, which confirms that all these observed effectiveness differences are the result of different fine-tuning procedures using in-domain data (i.e., robust04 or gov2). interestingly, even using all available data for robust04, the model only modestly improves over the zero-shot baseline. this finding is consistent with the recent work of @xcite , who eschew in-domain training data completely in the context of ranking with sequence-to-sequence models. ## conclusions this paper shows that, on two trec collections from different domains, the effectiveness of a finetuned bert-maxp model plateaus as the amount of available judgements increases. this suggests that the current sizes of trec test collections are sufficient for training with current bert architectures for document retrieval. we find that performing zero-shot learning by adapting a model trained on a different domain provides a strong baseline and can even be a substitute for in-domain finetuning under data-poor conditions. whether these results are due to limitations with existing datasets (e.g., their annotation schemes), ranking models (e.g., their inability to extract more signal), or the training procedure (e.g., hyperparameter settings to properly mix out-of-domain and in-domain data) remains an interesting open-research question. our findings present interesting guidance to practitioners: before embarking on any annotation effort in a document ranking task, it would be wise to first carefully plan out the amount of resources that are available. our experimental results show that a bit of data can be worse than none!