Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision (2021)

Abstract: The effectiveness of Neural Information Retrieval (Neu-IR) often depends on a large scale of in-domain relevance training signals, which are not always available in real-world ranking scenarios. To democratize the benefits of Neu-IR, this paper presents MetaAdaptRank, a domain adaptive learning method that generalizes Neu-IR models from label-rich source domains to few-shot target domains. Drawing on source-domain massive relevance supervision, MetaAdaptRank contrastively synthesizes a large number of weak supervision signals for target domains and meta-learns to reweight these synthetic “weak” data based on their benefits to the target-domain ranking accuracy of Neu-IR models. Experiments on three TREC benchmarks in the web, news, and biomedical domains show that MetaAdaptRank significantly improves the few-shot ranking accuracy of Neu-IR models. Further analyses indicate that MetaAdaptRank thrives from both its contrastive weak data synthesis and meta-reweighted data selection. The code and data of this paper can be obtained from https://github.com/thunlp/MetaAdaptRank .

## introduction text retrieval aims to rank documents to either directly satisfy users' search needs or find textual information for later processing components, e.g., question answering @xcite and fact verification @xcite . neural information retrieval (neu-ir) models have recently shown advanced results in many ranking scenarios where massive relevance labels or clickthrough data are available @xcite . the flip side is that the "data-hungry" nature of neu-ir models yields mixed results in few-shot ranking scenarios that suffer from the shortage of labeled data and implicit user feedback @xcite . on ranking benchmarks with only hundreds of labeled queries, there have been debates about whether neu-ir, even with billions of pre-trained parameters @xcite , really outperforms traditional ir techniques such as feature-based models and latent semantic indexing @xcite . in fact, many real-world ranking scenarios are fewshot, e.g., tail web queries that innately lack large supervision @xcite , applications with strong privacy constraints like personal and enterprise search @xcite , and domains where labeling requires professional expertise such as biomedical and legal search @xcite . to broaden the benefits of neu-ir to few-shot scenarios, we present an adaptive learning method metaadaptrank that meta-learns to adapt neu-ir models to target domains with synthetic weak supervision. for synthesizing weak supervision, we take inspiration from the work @xcite that generates related queries for unlabeled documents in a zero-shot way, but we generate discriminative queries based on contrastive pairs of relevant (positive) and irrelevant (negative) documents. by introducing the negative contrast, metaadaptrank can subtly capture the difference between documents to synthesize more ranking-aware weak supervision signals. given that synthetic weak supervision inevitably contains noises, metaadaptrank metalearns to reweight these synthetic weak data and trains neu-ir models to achieve the best accuracy on a small volume of target data. in this way, neural rankers can distinguish more useful synthetic weak supervision based on the similarity of the gradient directions of synthetic data and target data @xcite instead of manual heuristics or trialand-error data selection @xcite . we conduct experiments on three trec benchmarks, clueweb09, robust04, and trec-covid, which come from the web, news, and biomedi-cal domains, respectively. metaadaptrank significantly improves the few-shot ranking accuracy of neu-ir models across all benchmarks. we also empirically indicate that both contrastive weak data synthesis and meta-reweighted data selection contribute to metaadaptrank's effectiveness. compared to prior work @xcite , metaadaptrank not only synthesizes more informative queries and effective weak relevance signals but customizes more diverse and fine-grained weights on synthetic weak data to better adapt neural rankers to target few-shot domains. the effectiveness of neu-ir methods heavily relies on the end-to-end training with a large number of relevance supervision signals, e.g., relevance labels or user clicks. nevertheless, such supervision signals are often insufficient in many ranking scenarios. the less availability of relevance supervision pushes some neu-ir methods to freeze their embeddings to avoid overfitting @xcite . the powerful deep pre-trained language models, such as bert @xcite , also do not effectively alleviate the dependence of neu-ir on a large scale of relevance training signals. recent research even observes that bert-based neural rankers might require more training data than shallow neural ranking models @xcite . moreover, they may often be overly confident and more unstable in the learning process @xcite . a promising direction to alleviate the dependence of neu-ir models on large-scale relevance supervision is to leverage weak supervision signals that are noisy but available at mass quantity @xcite @xcite . through ir history, various weak supervision sources have been used to approximate querydocument relevance signals, e.g., pseudo relevance labels generated by unsupervised retrieval methods @xcite , and title-document pairs @xcite . recently, @xcite treat paired anchor texts and linked pages as weak relevance signals and propose a reinforcement-based data selection method reinfoselect, which learns to filter noisy anchor signals with trial-and-error policy gradients. despite their convincing results, anchor signals are only available in web domains. directly applying them to non-web domains may suffer from suboptimal outcomes due to domain gaps. to obtain weak supervision that adapts arbitrary domains, @xcite present a synthetic query generation method, which can be trained with source-domain relevance signals and applied on target-domain documents to generate related queries. more recently, a novel meta-learning technique has shown encouraging progress on solving data noises and label biases in computer vision @xcite @xcite and some nlp tasks @xcite . to the best of our knowledge, this novel technique has not been well utilized in information retrieval and synthetic supervision settings. ## methodology this section first recaps the preliminary of neu-ir and then introduces our proposed metaadaptrank. the framework of our method is shown in figure 1 . ## experimental methodology this section describes our experimental settings and implementation details. datasets. as shown in evaluation settings. we evaluate supervised ir methods through re-ranking the top 100 documents from the first-stage retrieval with five-fold cross-validation, consistent with prior work @xcite @xcite . the first-stage retrieval for clueweb09-b and robust04 is the sequential dependence model (sdm) @xcite released by @xcite , and the first-stage retrieval for trec-covid is bm25 @xcite well-tuned by anserini @xcite . metrics. ndcg@20 is used as the primary metric for all datasets. we also report err@20 for clueweb09-b and robust04, which is the same with prior work @xcite , and report p@20 for trec-covid. statistic significance is examined by permutation test with @xmath0.05. baselines. two groups of baselines are compared in our experiments, including traditional ir baselines and neural ir baselines. traditional ir baselines. following previous research @xcite , we compare four traditional ir methods in our experiments. they are two unsupervised methods, bm25 @xcite and sdm @xcite , and two learningto-rank (ltr) methods using bag-of-word features, ranksvm @xcite and coor-ascent (coordinate ascent) @xcite . neural ir baselines. we also compare seven neu-ir baselines that utilize different methodologies to train neural rankers. in our experiments, all neu-ir methods adopt the widely-used bert ranker @xcite , bert-firstp, which only uses the first paragraph of documents. the vanilla neural baseline only leverages the existing small-scale relevance labels of target datasets to train bert rankers, which is named few-shot supervision. we also compare bert rankers trained with two large-scale supervision sources: bing user click and ms marco. @xcite train bert rankers with 5 million user click logs in bing. we borrow their reported results because commercial logs are not publicly available. ms marco is a human supervision source @xcite , which provides over one million bing queries with relevance labels. four weak supervision methods are also compared. one baseline is title fitler, which treats filtered title-document pairs as weak supervision signals @xcite for training bert rankers @xcite . another two baselines are anchor and reinfoselect. anchor leverages 100k pairs of anchor texts and web pages to train bert rankers @xcite . reinfoselect first employs reinforcement learning to select these anchor signals @xcite and then trains bert rankers. the last baseline syncsup trains bert rankers with synthetic weak supervision data, which are synthesized based on the previous work @xcite . implementation details. this part introduces the implement details of our method and baselines. bert ranker. for our methods and all neu-ir baselines, we use the base version of bert @xcite on clueweb09-b and robust04, and pubmedbert (base) @xcite on trec-covid. we leverage the openmatch @xcite implementation and obtain the pre-trained weights from hugging face @xcite . for all neu-ir methods, we first use additional supervision sources such as weak supervision signals to train bert rankers (except for few-shot supervision); then fine-tune the bert rankers with the training folds of target datasets in the crossvalidation. following prior work @xcite , the ranking features ([cls] embeddings) of bert are combined with the first-stage retrieval scores using coor-ascent for clueweb09-b and robust04. we set the max input length to 512 and use adam optimizer with a learning rate of 2e-5 and a batch size of 8. ## evaluation results in this section, we present the evaluation results of metaadaptrank and conduct a series of analyses and case studies to study its effectiveness. ## conclusion this paper presents metaadaptrank, a domain adaption method for few-shot neu-ir with contrastive weak data synthesis and meta-reweighted data selection. contrastive synthesis generates informative queries and useful synthetic supervision signals. meta-learned weights form high-resolution channels between target labels and synthetic signals, providing robust and fine-grained data selection for synthetic weak supervision. both of them collaborate to significantly improve the neural ranking accuracy in various few-shot search scenarios.