C o RT : Complementary Rankings from Transformers (2021)

Abstract: Many recent approaches towards neural information retrieval mitigate their computational costs by using a multi-stage ranking pipeline. In the first stage, a number of potentially relevant candidates are retrieved using an efficient retrieval model such as BM25. Although BM25 has proven decent performance as a first-stage ranker, it tends to miss relevant passages. In this context we propose CoRT, a simple neural first-stage ranking model that leverages contextual representations from pretrained language models such as BERT to complement term-based ranking functions while causing no significant delay at query time. Using the MS MARCO dataset, we show that CoRT significantly increases the candidate recall by complementing BM25 with missing candidates. Consequently, we find subsequent re-rankers achieve superior results with less candidates. We further demonstrate that passage retrieval using CoRT can be realized with surprisingly low latencies.

## introduction the successful development of neural ranking models over the past few years has rapidly advanced state-of-the-art performance in information retrieval @xcite . one key aspect of the success is the exploitation of query-document interactions based on token representations from self-supervised language models (lms) @xcite @xcite . due to high computational effort, however, these interaction-focused approaches are limited to re-ranking scenarios and thus they depend on the effectiveness of first-stage ranking models for candidate retrieval. term-based retrieval models such as bm25 have proven decent performance in this task, but tend to miss relevant passages. in this context, we propose complementary rankings from transformers (cort), a simple neural first-stage ranking model that lever-ages contextual representations from transformerbased language models @xcite to complement term-based firststage rankings. cort optimizes an underlying text encoder towards representations that reflect relevance through vector similarity. the model is trained to act complementary to term-based retrieval by using passages from bm25 rankings as negative examples. we study the characteristics of cort with four types of experiments based on the ms marco dataset. first, we measure various ranking metrics and compare the results with first-stage ranking baselines and competitors. in course of this, we demonstrate the portion of relevant candidates that are added by cort. second, we combine the candidates from cort and bm25 with a state-of-the-art re-ranker based on bert @xcite and investigate how many candidates are needed to saturate the ranking quality. third, we train cort with various representation sizes and measure its impact on the first-stage ranking quality. fourth, we measure the retrieval latencies of cort with two retrieval modalities: a distributed exhaustive search on four gpus and an approximate search based on a graph-based nearest-neighbor index with pruning heuristics @xcite . finally, we build an exemplary end-to-end ranking pipeline using our first-stage ranking to demonstrate its efficiency. our contribution is a first-stage ranking framework with the potential to improve end-toend ranking pipelines by adding candidates that term-based retrieval models typically miss. with this it is possible to reduce the total number of re-ranking candidates without hurting end-to-end ranking quality. as a secondary contribution, we provide an open-source implementation foot_0 that enables other researchers to reproduce our results and test cort on other datasets. ## 2 background and related work in this section we describe key concepts of neural ranking and refer to related work. we then present neural first-stage ranking approaches that allow direct comparison with our results. ## proposed approach we describe a first-stage ranking model that acts as a complementary ranker to existing term-based retrieval models such as bm25. to achieve this, we make use of a transformer-based pretrained language model and its inherent ability to make use of token-level local interactions. its complementary behavior is further supported by negative sampling from bm25 rankings. ## experiments we present four experiments studying the ranking quality and recall of cort, the connection between the number of candidates and re-ranking effectiveness, the impact of the representation size e, and cort's retrieval latencies. finally, we outline a competitive end-to-end ranking setup with cort and a bert-based re-ranker. ## conclusion in this paper, we propose cort, a framework and neural first-stage ranking model that leverages contextual representations from transformer-based language models to complement term-based ranking functions. as a result, we observe decently increased recall measures and improved end-to-end ranking quality on the ms marco passage task. also, we are able to decrease the number of candidates for re-ranking without hurting the final performance. our further experiments reveal sweet spots for cort's representation size and the number of re-ranking candidates. we presented two deployment strategies for cort and measured their performances in terms of efficiency and effectiveness. finally, we demonstrate cort can be used with a simple bert-based re-ranker to create a competitive ranking pipeline.