Learning to Rank in the Age of M uppets: Effectiveness–Efficiency Tradeoffs in Multi-Stage Ranking (2021)

Abstract: It is well known that rerankers built on pretrained transformer models such as BERT have dramatically improved retrieval effectiveness in many tasks. However, these gains have come at substantial costs in terms of efficiency, as noted by many researchers. In this work, we show that it is possible to retain the benefits of transformer-based rerankers in a multi-stage reranking pipeline by first using feature-based learning-to-rank techniques to reduce the number of candidate documents under consideration without adversely affecting their quality in terms of recall. Applied to the MS MARCO passage and document ranking tasks, we are able to achieve the same level of effectiveness, but with up to 18× increase in efficiency. Furthermore, our techniques are orthogonal to other methods focused on accelerating transformer inference, and thus can be combined for even greater efficiency gains. A higher-level message from our work is that, even though pretrained transformers dominate the modern IR landscape, there are still important roles for “traditional” LTR techniques, and that we should not forget history.

## introduction pretrained transformers such as bert @xcite have dramatically increased retrieval effectiveness in many tasks across a multitude of domains @xcite . nevertheless, in a standard "retrieve-then-rerank" setup, the application of pretrained transformer-based rerankers incurs large computational costs and long query latencies, making those rerankers unrealistic for many real-world applications. for example, according to the colbert paper @xcite , reranking 1000 hits from the ms marco passage dataset takes 32.9 seconds per query. other researchers have noted the computational costs of transformer-based rankers (hofstätter and hanbury, * equal contribution 2019), and this realization has compelled the field to explore other approaches, for example, simplified models @xcite @xcite @xcite and learned dense representations @xcite . we are also motivated by the desire to reduce the computational costs of ranking with transformers, but from a different perspective. based on the observation that neural networks in general (and transformers in particular) have largely supplanted feature-based learning to rank (ltr) in modern information retrieval, we ask the question: what, if anything, does "traditional" feature-based learning to rank have to offer in the age of muppets? 1 the subtext of this question is that we, as a field, should not forget our own history. there are two obvious approaches to try and answer this question. the first is to simply consider transformer-based features (e.g., bert score, colbert score, etc.) as yet another feature within a learning-to-rank framework-for example, with gradient boosted decision trees @xcite . this is not the route that we take, because this approach has less bearing on our desire to increase the efficiency of transformer-based models. instead, we take the alternative approach of using learningto-rank techniques as a "filtering" stage in a multistage ranking architecture to reduce the number of candidates under consideration by bert. more concretely, we find that a design based on this idea achieves the same level of effectiveness as a standard retrieve-and-rerank approach using bert, but is up to 18× faster. other effectiveness-efficiency tradeoffs are possible, giving developers a rich design space to build systems tailored to different application scenarios. the contribution of this work is to demon-strate that by inserting a "filtering" learning-to-rank stage prior to bert-based reranking, we can control effectiveness-efficiency tradeoffs in a manner that makes deployments in real-world applications more practical. we emphasize that this work is orthogonal to other methods that directly attempt to accelerate inference, e.g., via knowledge distillation, early exits, model simplifications, etc. it is likely that our gains are cumulative with respect to these enhancements. ## experimental setup data we use the ms marco passage ranking dataset @xcite for training and testing. the training set contains ∼500k queries while the development and test sets contain ∼7k queries each. on average, each query has only one positive example; negative examples are taken from bm25 results that are not otherwise judged as relevant. since it is inefficient to use all the negative samples, we downsample to 20 negative examples per query and combine them with all the positive examples to arrive at the training data. we additionally test on the ms marco document ranking task @xcite in a zeroshot manner. for this, we segment each document into multiple passages as the neural models cannot process long documents. specifically, we use the sliding window strategy of @xcite , where the window length is ten sentences with a stride of five sentences. retrieval is performed at the passage level, and the document score is computed based on the highest relevance score among its passages. implementation we use anserini @xcite , an open-source ir toolkit built on lucene, to build the indexes and retrieve top-ranked candidate passages. for first-stage retrieval, we use bm25, with parameters (k 1 = 0.82 and @xmath0.68) based on the authors' recommendations to optimize for recall@1000. the retrieved candidate passages are then sent to feature extraction through anserini's python interface, pyserini @xcite ). 2 for our ltr module, we use the lambdamart algorithm implemented in lightgbm foot_2 as our model. the hyperparameters are tuned to achieve the highest recall@200 on the development set. specifically, num_leaves is 200, learning_rate is 0.1, min_data_in_leaf is 50, max_bin is 255. we fix early stopping patience to 200 and use up to 1000 trees. finally, we utilize batch processing and multithreading to accelerate the processing. this helps us leverage contemporary multi-core cpus. for the final-stage neural reranker, we experiment with bert-large and t5-base in the pygaggle library fine-tuned on the ms marco passage data. 4 we simply use checkpoints provided by the library, as our work is not specifically focused on final-stage neural reranking. previous evaluations @xcite @xcite have already verified that these two models serve as competitive baselines. we pad all the token sequences in the batch to have the same length and truncate them if their lengths exceed 512 tokens. latency measurements when measuring latency, we used two different servers. the latency of first-stage retrieval and the ltr filtering module is measured on a server equipped with 2 intel xeon platinum 8160 cpus. the index is located on a local ssd partition. the neural model latency is measured using a 6 core server with a single tesla v100 gpu. all latency measurements exclude the time to load data and models. component results are summed to yield end-to-end query latency, which we normalize into a speedup value when comparing different conditions. ## related work at a high level, the entire premise of our work is the point of multi-stage ranking, in that the architecture evolved to achieve a good balance between effectiveness and efficiency in end-to-end retrieval. motivated by the observation, dating back more than a decade, that effective techniques are often computationally expensive, multi-stage retrieval architectures control latency by applying expensive techniques over only the most promising candidates @xcite . this is often operationalized as optimizing for recall in the earlier stages of the pipeline. specifically in the context of transformers, multi-stage neural pipelines have been explored in the past by many researchers @xcite @xcite . the key difference in our work is the (re-)introduction of "traditional" feature-based learning-to-rank approaches alongside neural models. this aligns with our broader goal of investigating how learning to rank might contribute to modern retrieval approaches dominated by neural models. the computational costs associated with ranking using pretrained transformers can be reduced in various ways. we can accelerate inference using smaller or simpler models. @xcite use distillation to transfer knowledge captured in a larger model into a smaller model, achieving substantial speedups with minimal effectiveness loss. @xcite propose a simpler transformer model to capture contextual information that trades effectiveness for much faster inference. additional examples of this approach include @xcite and @xcite . an alternative is to introduce early-exit optimizations, as in @xcite and @xcite . further speedups can be gained by making modifications to the backbone transformer model, as in @xcite . the key point is that our proposed ltr filtering module achieves speedups in a manner that is orthogonal to the methods discussed here, which focus on directly accelerating transformer inference. thus, these approaches can be combined with our method for even greater efficiency gains. ## conclusions the "retrieve-then-rerank" approach with transformers has been demonstrated to be effective in many ir tasks, but poor efficiency makes it less attractive for real-world applications. our goal is to increase the efficiency of the entire pipeline but at the same time maintain the same level of effectiveness: this is achieved by a feature-based learning-to-rank module that filters candidates prior to neural reranking. on the ms marco passage ranking task, we observe up to 18× speedup without degradation in terms of mrr@10. in a zeroshot setting on the ms marco document ranking task, we can achieve 3× speedup. these results demonstrate that in this age of muppets dominated by transformers and other neural models, learningto-rank techniques can still be useful. despite ltr "falling out of fashion", we should not "throw the baby out with the bath water". one key point that bears emphasis, and one promising direction for future research is that our work can be combined with other approaches that directly accelerate inference in neural models. we expect speedups to be cumulative since we tackle efficiency issues from an orthogonal perspective.