Collective Relevance Labeling for Passage Retrieval (2022)

Abstract: Deep learning for Information Retrieval (IR) requires a large amount of high-quality query-document relevance labels, but such labels are inherently sparse. Label smoothing redistributes some observed probability mass over unobserved instances, often uniformly, uninformed of the true distribution. In contrast, we propose knowledge distillation for informed labeling, without incurring high computation overheads at evaluation time. Our contribution is designing a simple but efficient teacher model which utilizes collective knowledge, to outperform state-of-the-arts distilled from a more complex teacher model. Specifically, we train up to \times8 faster than the state-of-the-art teacher, while distilling the rankings better. Our code is publicly available at https://github.com/jihyukkim-nlp/CollectiveKD .

## introduction facilitated by recent developments in pre-trained language models (plm) such as bert @xcite , neural ranking models have seen significant improvements in effectiveness @xcite . neural ranking models can be categorized into two major groups: cross-encoders and bi-encoders, which jointly or separately encode a query q and a passage p, respectively. while the former shows stronger performance, the latter makes the passage representations indexable and thus enables efficient retrieval supported by approximated nearest neighbor (ann) search, e.g., faiss @xcite . in this work, targeting web-scale retrieval, we follow the bi-encoder design and adopt the state-of-the-art bi-encoder, colbert @xcite as our target retriever. while label supervision has a critical role in training bi-encoders, only incomplete labels are available in benchmark training datasets due to the prohibitive cost of exhaustive human annotation of large-scale passage corpora. as a prominent example, in msmarco @xcite training dataset, the number of labeled relevant passages per query averages only 1.1, among 8.8m passages. in contrast, in trec-dl 2019 @xcite ) which provides complete annotations for a select number of queries on the same passage collection, the same value is 58.2, indicating that significant amounts of relevant passages can remain unlabeled in msmarco. meanwhile, we stress that the problem of incomplete labels observed from benchmark datasets gets worse in reallife retrieval tasks, where new relevant documents are constantly added without annotation. existing work addressing incomplete labels can be categorized into uninformed smoothing and informed labeling. uninformed smoothing redistributes the observed probability mass uniformly over the unobserved, and has been shown to improve model calibration and prediction @xcite . however, given diverse topics in the passage corpora, uniform smoothing does not accurately reflect the true distribution. alternatively, informed labeling distills knowledge from a trained model, evaluating the relevance of a q-p pair, to assign a higher probability to an unlabeled pair with a higher estimated relevance, also known as knowledge distillation (kd) @xcite . to augment capacity without sacrificing efficiency, our distinction is utilizing a set of relevant passages from the bi-encoder teacher as collective knowledge. to illustrate, in we validate the effectiveness of our proposed collective bi-encoder for kd, comparing retrieval efficacy with existing models on the trec and msmarco datasets. while achieving the state-of-the-art performance, we train up to ×8 times faster than existing kd that uses cross-encoder teachers. ## approach taking the state-of-the-art bi-encoder colbert as our target retriever ( §2.1), our goal is to tackle the problem of incomplete labels during training ( §2.2). to do so, we devise a stronger colbert teacher that leverages collective knowledge to refine relevance labels that are transferred to the colbert student via kd ( §2.3). ## conclusion and future work we study collective relevance labeling to overcome incomplete labels in passage retrieval. our approach bypasses the computational overhead associated with prf, leading to a state-of-the-art student retriever without sacrificing efficiency. we validate the effectiveness of our method over existing approaches on the msmarco and trec-dl passage ranking datasets. as future work, we consider zero-shot transfer. while our collective self-kd effectively distills knowledge, one requirement we have is to bootstrap the collective knowledge with a good initial ranking model. as it usually requires some form of label supervision to train such a model, the case where no initial supervision is available, may be considered a limitation of our method. we believe bootstrapping an effective self-kd without any supervision is a promising direction.