Pretrained Transformers for Text Ranking: BERT and Beyond (2021)

Abstract: The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query for a particular task. Although the most common formulation of text ranking is search, instances of the task can also be found in many text processing applications. This tutorial provides an overview of text ranking with neural network architectures known as transformers, of which BERT (Bidirectional Encoder Representations from Transformers) is the best-known example. These models produce high quality results across many domains, tasks, and settings. This tutorial, which is based on the preprint of a forthcoming book to be published by Morgan and & Claypool under the Synthesis Lectures on Human Language Technologies series, provides an overview of existing work as a single point of entry for practitioners who wish to deploy transformers for text ranking in real-world applications and researchers who wish to pursue work in this area. We cover a wide range of techniques, grouped into two categories: transformer models that perform reranking in multi-stage ranking architectures and learned dense representations that perform ranking directly.

## multi-stage ranking architectures the most straightforward application of transformers to text ranking is to convert the task into a text classification problem, and then sort the texts to be ranked based on the probability that each item belongs to the relevant class. the first application of bert to text ranking, by @xcite , used bert in exactly this manner. this relevance classification approach is usually deployed in a module that reranks candidate texts from an initial keyword search engine. one key limitation of bert is its inability to handle long input sequences and hence difficulty in ranking texts beyond a certain length (e.g., "full-length" documents such as news articles). this limitation is addressed by a number of models @xcite @xcite @xcite , and a simple retrieve-then-rerank approach can be elaborated into a multi-stage architecture with reranker pipelines @xcite @xcite ) that balance effectiveness and efficiency. on top of multi-stage ranking architectures, researchers have proposed additional innovations, including query expansion @xcite , document expansion @xcite and term importance prediction @xcite . a natural question that arises is, "what's beyond bert?" we describe efforts to build ranking models that are faster (i.e., lower inference latency), that are better (i.e., higher ranking effectiveness), or that manifest interesting tradeoffs between effectiveness and efficiency. these include ranking models that leverage bert variants @xcite , exploit knowledge distillation to train more compact student models @xcite , and other transformer architectures, including groundup redesign efforts @xcite and adapting pretrained sequence-tosequence models @xcite . these discussions set up a natural transition to ranking based on dense representations, the other main category of approaches we cover. ## learned dense representations arguably, the single biggest benefit brought about by modern deep learning techniques to text ranking is the move away from sparse signals, mostly limited to exact matches, to dense representations that are able to capture semantic matches to better model relevance. the potential of continuous dense representations for natural language analysis was first demonstrated nearly a decade ago with word embeddings on word analogy tasks @xcite . as soon as researchers tried to build representations for any larger spans of text: phrases, sentences, paragraphs, and documents, the same issues that arise in text ranking come into focus. in fact, ranking with dense representations predates bert by many years @xcite @xcite @xcite . in the context of transformers, the general setup of ranking with dense representations involves learning transformer-based encoders that convert queries and texts into dense, fixed-size vectors. in the simplest approach, ranking becomes the problem of approximate nearest neighbor (ann) search based on some simple metric such as cosine similarity @xcite @xcite @xcite @xcite @xcite @xcite . however, recognizing that accurate ranking cannot be captured via simple metrics, researchers have explored using more complex machinery to compare dense representations @xcite . here, as with multi-stage ranking architectures, limitations on text length and effectiveness-efficiency tradeoffs are important considerations. it becomes increasingly difficult to accurately capture the semantics of longer texts with fixed-sized representations, and increasingly complex comparison architectures increase latency and may necessitate reranking designs. ## looking ahead learned dense representations complement sparse (bag-of-words) term-based representations central to keyword search techniques that have dominated the landscape for more than half a century. together, hybrid multi-stage approaches (e.g., combining both ranking and reranking) present a promising future direction. despite the excitement in directly ranking with dense learned representations, we anticipate that reranking transformers will remain important in the future. for one, results from dense retrieval can usually be reranked to achieve even higher effectiveness. at a high level, there are three current approaches: apply existing transformer models with minimal modifications, adapt existing transformer models, perhaps adding additional architectural elements, and redesign transformer-based architectures from scratch. which approach will prove to be most effective? the jury's still out. related, in nlp we see that the gpt family @xcite continues to push the frontier of larger models, more compute, and more data. for text ranking, is the simple answer to build bigger models? probably not, since ranking has important differences with many traditional nlp tasks. but if not, what are the evolving roles of zeroshot learning, distant supervision, transfer learning, domain adaptation, data augmentation, and taskspecific fine-tuning? this remains an interesting open research question. while there are aspects of text ranking with pretrained transformers that are well understood, many promising directions await further exploration. looking ahead, we anticipate many more exciting developments!