In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage ranking architectures and learned dense representations that attempt to perform ranking directly. There are numerous examples that fall into the first category, including approaches based on relevance classification, evidence aggregation from multiple segments of text, corpus analysis, and sequence-to-sequence models. While the second category of approaches is less well studied, representation learning with transformers is an emerging and exciting direction that is bound to attract more attention moving forward. There are two themes that pervade our survey: techniques for handling long documents, beyond the typical sentence-by-sentence processing approaches used in NLP, and techniques for addressing the tradeoff between effectiveness (result quality) and efficiency (query latency). 
The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query for a particular task. The most common formulation of text ranking is search, where the search engine (also called the retrieval system) produces a ranked list of texts (web pages, scientific papers, news articles, tweets, etc.) ordered by estimated relevance with respect to the user's query. In this context, relevant texts are those that are "about" the topic of the user's request and address the user's information need. Information retrieval (IR) researchers call this the ad hoc retrieval problem. 1
With keyword search, also called keyword querying (for example, on the web), the user typically types a few query terms into a search box (for example, in a browser) and gets back results containing representations of the ranked texts. These results are called ranked lists, hit lists, hits, "ten blue links", 2 or search engine results pages (SERPs). The representations of the ranked texts typically comprise the title, associated metadata, extracts from the texts themselves (for example, a keywordin-context summary where the user's query terms are highlighted), as well as links to the original sources. While there are plenty of examples of text ranking problems (see Section 1.1), this particular scenario is ubiquitous and undoubtedly familiar to all readers.
This survey provides an overview of text ranking with a family of neural network architecture known as transformers, of which BERT (Bidirectional Encoder Representations from Transformers) , an invention of Google, is the best-known example. BERT has, without exaggeration, revolutionized the fields of natural language processing (NLP) and information retrieval (IR), and more broadly, human language technologies (HLT), a catch-all term that includes technologies that process, analyze, and otherwise manipulate (human) language data. There are few endeavours involving the automatic processing of natural language that remain untouched by BERT. 3 In the context of text ranking, BERT provides results that are undoubtedly superior in quality than what came before. This is a robust and widely reproduced empirical result, across many text ranking tasks, domains, and problem formulations.
A casual skim through paper titles in recent proceedings from NLP and IR conferences will leave the reader without a doubt as to the extent of the "BERT craze" and how much it has come to dominate the current research landscape. However, the impact of BERT, and more generally, transformers, has not been limited to academic research. In October 2019, a Google blog post 4 confirmed that the company had improved search "by applying BERT models to both ranking and featured snippets". Ranking refers to "ten blue links" and corresponds to most users' understanding of web search; "feature snippets" represent essentially a question answering feature 5 (see additional discussion in Section 1.1). Not to be outdone, in November 2019, a Microsoft blog post 6 reported that "starting from April of this year, we used large transformer models to deliver the largest quality improvements to our Bing customers in the past year."
Beyond search, transformers dominate approaches to machine translation, which is the automatic translation of natural language text (and sometimes even speech directly) from one human language to another, for example, from English to French. Blog posts by both Facebook 7 and Google 8 tout the effectiveness of transformer-based models. Of course, these are just the high-profile announcements.
No doubt many organizations-from startups to Fortune 500 companies, from those in the technology sector to those in financial services and beyond-have already or are planning to deploy BERT (or one of its siblings or intellectual decedents) in production.
Transformers were first presented in June, 2017 [Vaswani et al., 2017] and BERT was unveiled in October, 2018. 9 Although both are relatively recent inventions, we believe that there is a sufficient body of research such that the broad contours of how to apply transformers effectively for text ranking have begun to emerge, from high-level design choices to low-level implementation details. The "core" aspects of how BERT is applied-for example, as a relevance classifier-is relatively mature.
Many of the techniques we present have been applied in many domains, tasks, and settings, and the improvements brought about by BERT are usually substantial and robust. It is our goal to provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply BERT to text ranking problems and researchers who wish to pursue work in this area.
Like nearly all scientific advances, BERT was not developed in a vacuum, but built on several previous innovations, most notably the transformer architecture itself [Vaswani et al., 2017] , ULMFiT [Howard and Ruder, 2018] , ELMo (Embeddings from Language Models) [Peters et al., 2018] , and GPT (Generative Pretrained Transformer) [Radford et al., 2018] (see additional discussions in Section 3.1). While it is important to recognize previous work, BERT is distinguished in that it represented the first model where the crucial ingredients-most importantly, transformers architectures and selfsupervised pretraining-"came together" successfully to yield tremendous leaps in effectiveness on a broad range of natural language processing tasks. The commendable decision by Google to open-source BERT model code and release pretrained models supported widespread replication of the impressive results reported by the authors and supercharged subsequent innovations by providing a replicable foundation to build on. BERT, in turn, has spawned a stampede of other models that can be viewed as variations on its main theme. These include XLNet [Yang et al., 2019f] , RoBERTa [Liu et al., 2019c] , ALBERT [Lan et al., 2020] , Megatron-LM [Shoeybi et al., 2019] , ELECTRA [Clark et al., 2020] , Reformer [Kitaev et al., 2020] , Big Bird [Zaheer et al., 2020] , and many more.
Although a major focus of this survey is BERT, many of the same techniques we discuss can (and have been) applied to its descendants and relatives as well, and BERT is often incorporated as part of a larger neural model. While BERT is no doubt the "star of the show", there are many exciting developments beyond BERT being explored right now: the application of sequence-tosequence transformers, transformer variants that yield more efficient inference, ground-up redesigns of transformer architectures, and representational learning with transformers-just to name a few (all of which we will cover). The diversity of research directions being actively pursued explains our choice for the subtitle of this survey ("BERT and Beyond"). In addition to synthesizing the more "mature" applications of BERT for text ranking, we discuss where the gaps in knowledge lie and highlight where we think the field is going.
Let us begin!
While our survey opens with search (specifically, ad hoc retrieval) as the motivating scenario due to the ubiquity of search engines, text ranking appears in many other guises. Beyond typing keywords into a search box and getting back "ten blue links", examples of text ranking abound in scenarios where users desire access to relevant textual information, in a broader sense.
Consider the following examples: 9 The nature of academic publishing today means that preprints are often available (e.g., on arXiv) several months before the formal publication of the work in a peer-reviewed venue (which is becoming increasingly a formality). For example, the BERT paper was first posted on arXiv in October 2018, but was not published in NAACL 2019 (a top conference in NLP) until June 2019. Throughout this survey, we refer to the earliest known preprint publication date, since that is the date when a work becomes "public" and available for other researchers to examine, critique, and extend. For example, the earliest use of BERT for text ranking was reported in January 2019 [Nogueira and Cho, 2019] , a scant three months after the appearance of the original BERT preprint and well before the peer-reviewed NAACL publication. The rapid pace of progress in NLP, IR, and other areas of computer science today means that by the time an innovation formally appears in a peer-reviewed venue, the work is often already "old news", and in some cases, as with BERT, the innovation had already become widely adopted. In general, we make an effort to cite the peer-reviewed version of a publication unless there is some specific reason otherwise, e.g., to establish precedence.
Question Answering (QA). In question answering, a system aims to identify a span of text that directly answers the user's question, instead of returning a list of documents (or even extracts) that the user must then manually peruse. Although the history of question answering systems dates back to at least the 1960s [Simmons, 1965] , modern approaches trace their roots to work that began in the late 1990s [Voorhees, 2001] . In the web context, question answering capabilities are often exposed in search engines through so-called "infoboxes" that appear before (or to the right of) the main search results. In the context of a voice-capable intelligent agent such as Siri or Alexa, answers to user questions are directly synthesized using text-to-speech technology. Most QA architectures break the challenge into two steps: first, select passages of text from a potentially large corpus that are likely to contain answers, and second, apply answer extraction techniques to identify the answer spans. 10 The first step is a ranking problem, and the second step is often formulated as a ranking problem. That is, selecting among answer candidates (which are spans of text themselves) can be formulated as another instance of text ranking.
Community Question Answering (CQA). Users sometimes search for answers not by attempting to find relevant information directly, but by locating another user who has asked the same or similar question, for example, in a frequently-asked questions (FAQ) list or in an online forum such as Quora or Stack Overflow. Answers to those questions usually address the user's information need. This mode of searching, which dates back to the late 1990s [Burke et al., 1997] , is known as community question answering (CQA) [Srba and Bielikova, 2016] . Although it differs from traditional keywordbased querying, CQA is nevertheless a text ranking problem. One standard approach formulates the problem as estimating semantic similarity between two pieces of texts-more specifically, if two natural language questions are paraphrases of each other. A candidate list of questions (for example, based on keyword search) are then sorted by the estimated degree of "paraphrase similarity" (for example, the output of a machine-learned model) and returned to the user as results.
Information Filtering. In search, queries are posed against a (mostly) static collection of texts. Filtering considers the opposite scenario where a (mostly) static query is posed against a stream of texts. Two examples of this mode of information seeking might be familiar to readers: push notifications that are sent to a user's mobile device whenever some content of interest is published (could be a news story or a social media post); and, in a scholarly context, email digests that are sent to users whenever a paper that matches the user's interest is published (a feature available in Google Scholar today). Not surprisingly, information filtering has a long history, dating back to the 1960s, when it was called "selective dissemination of information" (SDI); see Housman and Kaskela [1970] for a survey of early systems. The most recent incarnation of this idea is "real-time summarization" in the context of social media posts on Twitter, with several community-wide evaluations focused on notification systems that inform users in real time about relevant content as it is being generated [Lin et al., 2016b] . Before that, document filtering was explored in the context of the TREC Filtering Tracks, which ran from 1995 [Lewis, 1995] to 2002 [Robertson and Soboroff, 2002] , and the general research area of topic detection and tracking, also known as TDT [Allan, 2002] . The relationship between search and filtering has been noted for decades: Belkin and Croft [1992] famously argued that they represented "two sides of the same coin". The same ranking models that assign relevance scores in information retrieval can be adapted for information filtering, and thus information filtering can be considered a text ranking problem.
Text Recommendation. When a system is displaying a search result, it might suggest other texts that may be of interest to the user, for example, to assist in browsing [Smucker and Allan, 2006] . This is frequently encountered on news sites, where related articles of interest might provide background knowledge or pointers to related news stories [Soboroff et al., 2018] . In the context of searching the scientific literature, the system might suggest papers that are similar in content; an example of this feature is implemented in the PubMed search engine, which provides access to the scientific literature in the life sciences [Lin and Wilbur, 2007] . Although such features can be characterized as recommendations, the problem involves text ranking.
Text Ranking as Input to Downstream Modules. The output of text ranking may not be intended for user consumption, but is rather meant to feed downstream processing components: for example, an information extraction module to identify key entities and relations [Gaizauskas and Robertson, 1997 ], a summarization module that attempts to synthesize information from multiple sources with respect to an information need [Dang, 2005] , a clustering module that organizes texts based on content similarity [Vadrevu et al., 2011] , or a browsing interface for exploration and discovery [Sadler, 2009] . Even in cases where a ranked list of results is not directly presented to the user, text ranking may still form an important component technology in a larger system.
We can broadly characterize ad hoc retrieval, question answering, and the different scenarios described above as "information access"-a term we use to refer to these technologies collectively. Text ranking is without a doubt an important component of information access.
However, beyond information access, examples of text ranking abound in natural language processing. For example:
Distant Supervision and Data Augmentation. Training data form a crucial ingredient of NLP approaches based on supervised machine learning. All things being equal, the more data the better, 11 and so there is a never-ending quest for practitioners and researchers to acquire more, more, and more! Supervised learning requires training examples that have been annotated for the specific task, typically by humans, which is a labor-intensive process. For example, to train a sentiment classifier, we have to somehow acquire a corpus of texts in which each instance has been labeled with its sentiment (e.g., positive or negative). There are natural limits to the amount of data that can be acquired via human annotation: in the sentiment analysis example, we can automatically harvest various online sources that have "star ratings" associated with texts (e.g., reviews), but even these labels are ultimately generated by humans. This is a form of crowdsourcing, and merely shifts the source of the labeling effort, but does not change the fundamental need for human annotation.
Researchers have extensively explored many techniques to overcome the data bottleneck in supervised machine learning. At a high level, distant supervision and data augmentation represent two successful approaches, although in practice they are closely related. Distant supervision involves training models using low-quality "weakly" labeled examples that are gathered using heuristics and other simple but noisy techniques. One simple example is to assume that all emails mentioning Viagra are spam for training a spam classifier; obviously, there are "legitimate" non-spam emails that use the term, but the heuristic may be a reasonable way to build an initial classifier . We give this example because it is easy to convey, but the general idea of using heuristics to automatically gather training examples to train a classifier in NLP dates back to Yarowsky [1995] , in the context of word sense disambiguation. 12
Data augmentation refers to techniques that exploit a set of training examples to gather or create additional training examples. For example, given a corpus of English sentences, we could translate them automatically using a machine translation (MT) system, say, into French, and then translate those sentences back into English (this is called back-translation). 13 With a good MT system, the resulting sentences are likely paraphrases of the original sentence, and using this technique we can automatically increase the quantity and diversity of the training examples that a model is exposed to.
Text ranking lies at heart of many distant supervision and data augmentation techniques for natural language processing. We illustrate with relation extraction, which is the task of identifying and extracting relationships in natural language text. For example, from the sentence "Albert Einstein was born in Ulm, in the Kingdom of Württemberg in the German Empire, on 14 March 1879", a system could automatically extract the relation birthdate(Albert Einstein, 1879/03/14); these are referred to as "tuples" or extracted facts. Relations usually draw from a relatively constrained ranking formulation can be augmented by attempts to perform joint inference to resolve cases where the same entity is identified as the most likely filler of more than one slot; for example, resolving the case where a model (independently) identifies "John" erroneously as both the most likely buyer and the most likely seller (which is semantically incoherent). Although the candidate entities are short natural language phrases, they can be augmented with a number of features, in which case the problem begins to look like ranking in a vector space model. While the number of entities to be ranked is not usually very big, what's important is the wealth of evidence (i.e., different features) that is leveraged to estimate the probability that an entity fills a role, which isn't very different from relevance classification (see Section 3.2).
Another problem that lends itself naturally to a ranking formulation is entity linking, where the task is to resolve an entity with respect to an external knowledge source such as Wikidata [Vrandečić and Krötzsch, 2014] . For example, in a passage of text that mentions Adam Smith, which exact person is being referenced? Is it the famous 18th century Scottish economist and moral philosopher, or one of the lessor-known individuals that share the same name? An entity linking system "links" the instance of the entity mention (in a piece of text) to a unique id in the knowledge source: the Scottish economist has the unique id of Q9381, 16 while the other individuals have different ids. Entity linking can be formulated as a ranking problem, where candidates from the knowledge source are ranked in terms of their likelihood of being the actual referent of a particular mention [Shen et al., 2015] . This is an instance of text ranking because these candidates are usually associated with textual descriptions-for example, a short biography of the individual-which forms crucial evidence. Here, the "query" is the entity to be linked, represented not only by its surface form (i.e., the mention string), but also the context in which the entity appears. For example, if the text discusses the Wealth of Nations, it's likely referencing the famous Scot.
Yet another example of text ranking in a natural language task that involves selecting from competing hypotheses is the problem of fact verification [Thorne et al., 2018] , for example, to combat the spread of misinformation online. Verifying the veracity of a claim requires fetching supporting evidence from a possibly large corpus and assessing the credibility of those sources. The first step of gathering possible supporting evidence is a text ranking problem. Here, the hypothesis space is quite large (passages from an arbitrarily large corpus), and thus text ranking plays a critical role. In the same vein, for systems that engage in or assist in human dialogue, such as intelligent agents or "chatbots", one common approach to generating responses (beyond question answering and information access discussed above) is to retrieve possible responses from a corpus (and then perhaps modifying it) [Henderson et al., 2017 , Roller et al., 2020 ]. Here, the task is to rank possible responses with respect to their appropriateness.
The point of this discussion is that while search is perhaps the most visible instance of the text ranking problem, there are manifestations everywhere-not only in information retrieval but also natural language processing.
This exposition also explains our rationale in intentionally using the term "text ranking" throughout this survey, as opposed to the more popular term "document ranking". In many applications, the "atomic unit" of text to be ranked is not a document, but rather a sentence, a paragraph, or even a tweet; see Section 2.1 for more details.
To better appreciate how BERT and transformers have revolutionized text ranking, it is first necessary to understand "how we got here". We turn our attention to this next in a brief exposition of important developments in information retrieval over the last three quarters of a century.
The vision of exploiting computing machines for information access is nearly as old as the invention of computing machines themselves, long before computer science emerged as a coherent discipline.
The earliest motivation for developing information access technologies was to cope with the explosion of scientific publications in the years immediately following World War II. 17 Vannevar Bush's oftencited essay in The Atlantic in July 1945, titled "As We May Think", predicted technologies in a 16 https://www.wikidata.org/wiki/Q9381 17 People have been complaining about there being more information than can be consumed since the invention of the printing press. hypothetical machine called the "memex" that we might recognize today as personal computers, hypertext and the web, and online encyclopedias. A clearer description of what we might more easily identify today as a search engine is provided by Holmstrom [1948] , although discussed in terms of punch-card technology!
Although the need for machines to improve information access was identified as early as the mid-1940s, interestingly, the conception of text ranking was still a decade away. Libraries, of course, have existed for millennia, and the earliest formulations of search were dominated by the automation of what human librarians had been doing for centuries: matching based on human-extracted descriptors of content stored on physical punch-card representations of the texts to be searched (books, scientific articles, etc.). These descriptors (also known as "index terms") were usually assigned by human subject matter experts (or at least trained human indexers) and typically drawn from thesauri, "subject headings", or "controlled vocabularies"-that is, a predefined vocabulary. This process was known as "indexing"-the original sense of the activity involved humans, and is quite foreign to modern notions that imply automated processing-or is sometimes referred to as "abstracting". 18 Issuing queries to search content required librarians (or at least trained individuals) to translate a searcher's information need into these same descriptors; search occurs by matching these descriptors in a boolean fashion (hence, no ranking).
As a (radical at the time) departure from this human-indexing approach, Luhn [1958] proposed considering "statistical information derived from word frequency and distribution . . . to compute a relative measure of signifcance", thus leading to "auto-abstracts". He described a precursor of what we would recognize today as tf-idf weighting (that is, term weights based on term frequency and inverse document frequency). However, Luhn neither implemented nor evaluated any of the techniques he described.
A clearer articulation of text ranking is presented in Maron and Kuhns [1960] , who characterized the information retrieval problem (although they didn't use these words) as receiving requests from the user and "to provide as an output an ordered list of those documents which most probably satisfy the information needs of the user". They proposed that index terms ("tags") be weighted according to the probability that a user desiring information contained in a particular document using that term in a query. Today, we might call this query likelihood. The paper also described the idea of a "relevance number" for each document, "which is a measure of the probability that the document will satisfy the given request". Today, we would call these retrieval scores. Beyond laying out these foundational concepts, Maron and Kuhns described experiments to test their ideas. We might take for granted today the idea that automatically extracted terms from a document can serve as descriptors or index terms for describing the contents of those documents, but this was an important conceptual leap in the development of information retrieval.
Throughout the 1960s and 1970s, the community debated the merits of "automatic content analysis" (see, for example, Salton [1968] ) vs. "traditional" human-based indexing. Salton [1972] described a notable evaluation comparing the SMART retrieval system based on the vector space model with MEDLARS, which is the precursor to MEDLINE and PubMed for searching literature in the life sciences (still used today), but at the time was based on human indexing. SMART was shown to produce higher-quality results, and Salton concluded "that no technical justification exists for maintaining controlled, manual indexing in operational retrieval environment".
The mode of information access we take for granted today-based on ranking automatically constructed representations of documents and queries-gradually gained acceptance, although the history of information retrieval showed this to be an uphill battle. Writing about the early history of information retrieval, Harman [2019] goes as far as to call these "indexing wars": the battle between human-derived and automatically-generated index terms. This is somewhat reminiscent of the rulebased vs. statistical NLP "wars" that raged beginning in the late 1980s and into the 1990s, and goes to show how foundational shifts in thinking are often initially met with resistance. Thomas Kuhn would surely find both these two cases to be great examples supporting his views on the structure of scientific revolutions [Kuhn, 1962] .
Bringing all the major ideas together, Salton et al. [1975] is frequently cited for the proposal of the vector space model, in which documents and queries are both represented as "bags of words" using sparse vectors according to some term weighting scheme (tf-idf in this case), where document-query similarity is computed in terms of cosine similarity (or, more generally, inner products). However, this development did not happen all at once, but represented innovations that gradually accumulated over the two preceding decades. For additional details about early historical developments in information retrieval, we refer the reader to Harman [2019] .
For the purposes of establishing a clear contrast with neural network models, the most salient feature of all approaches up to this point in history is their reliance exclusively on what we would call today exact term matching-that is, terms from documents and terms from queries had to match exactly to contribute to a ranking or a relevance score. Since systems typically perform stemming-that is, the elimination of suffixes (in English)-matching occurs after terms have been normalized to some extent (for example, stemming would ensure that "dog" matches "dogs").
Nevertheless, with techniques based on exact term matching, a scoring function between a query q and a document d could be written as:
where f is some function of a term and its associated statistics, the two most important of which are term frequency (how many times a term occurs in a document) and document frequency (the number of documents that contain at least once instance of the term). It is from these two basic statistics that we derive the ubiquitous scoring function tf-idf, which stands for term frequency, inverse document frequency.
A major thread of research in the 1980s and into the 1990s was the exploration of different term weighting schemes in the vector space model [Salton and Buckley, 1988a] , based on easily computed statistics such as term frequency, document frequency, document length, etc. One of the most successful of these methods, Okapi BM25 [Robertson et al., 1994 , Crestani et al., 1999 , Robertson and Zaragoza, 2009 , still provides the starting point of many text ranking approaches today, both in academic research as well as commercial systems. 19 Given the importance of BM25, the exact scoring function is worth repeating, to illustrate what a ranking model based on exact term matching looks like. The relevance score of a document d with respect to a query q is defined as:
As BM25 is based on exact term matching, the score is derived from a sum of contributions from each query term that appears in the document. In more detail:
• The first component of the summation (the log term) is the idf or inverse document frequency:
N is the total number of documents in the corpus, and df(t) is the number of documents that term t appears in (i.e., its document frequency).
• In the second component of the summation, tf(t, d) represents the number of times term t appears in document d (i.e., its term frequency). The expression in the denominator involving b is responsible for performing length normalization, since collections usually have documents that differ in length: l d is the length of document d while L is the average document length across all documents in the collection.
Finally, k 1 and b are free parameters. Note that the original formulation by Robertson et al. [1994] includes additional scoring components with parameters k 2 and k 3 , but they are rarely used and typically omitted from modern implementations. In addition to the original scoring function described above, there are number of variants that have been discussed in the literature, including the one implemented in the popular open-source Lucene search library; see Section 2.6 for more details.
While term weighting schemes can model term importance (sometimes called "salience") based on statistical properties of the texts, exact match techniques are fundamentally powerless in cases where terms from the query and the texts don't match at all. This happens quite frequently, when searchers use different terms to describe their information needs than what authors of the relevant documents used. 20 One way of thinking about search is that a user is trying to guess the terms (i.e., posed as the query) that authors of relevant texts would have used when they wrote the text (see additional discussion in Section 2.2). We're looking for a "tragic love story" but Shakespeare writes of "star-crossed lovers". To provide a less poetic, but more practical example, what we call "information filtering" today was known as "selective dissemination of information (SDI)" in the 1960s (see Section 1.1). Imagine the difficulty we would face trying to conduct a thorough literature review without knowing the relationship between these key terms. It would be easy to (falsely) conclude that no prior work exists! These are just two examples of the "vocabulary mismatch problem" [Furnas et al., 1987] , and represents a fundamental challenge in information retrieval. There are three general approaches to tackling this challenge: enrich query representations to better match document representations, enrich document representations to better match query representations, and attempts to go beyond exact term matching:
• Enriching query representations. One obvious approach to bridge the gap between query and document terms is to enrich query representations with query expansion techniques. In relevance feedback, the representation of the user's query is augmented with terms derived from documents that are known to be relevant (for example, documents that have been presented to the user and that the user has indicated is relevant): two popular formulations are based on the vector space model [Rocchio, 1971] and the probabilistic retrieval framework [Robertson and Spark Jones, 1976] . In pseudo-relevance feedback [Croft and Harper, 1979] , also called "blind" relevance feedback, top-ranking documents are simply assumed to be relevant, thus also providing a source for additional query terms. Query expansion techniques, however, do not need to involve relevance feedback: examples include Xu and Croft [2000] , who introduced global techniques that identify word relations from the entire collection as possible expansion terms (this occurs in a corpus preprocessing step, independent of any queries), and Voorhees [1994] , who experimented with query expansion using lexical-semantic relations from WordNet [Miller, 1995] . A useful distinction when discussing query expansion techniques is the dichotomy between pre-retrieval techniques, where expansion terms can be computed without examining any documents from the collection, and post-retrieval techniques, which are based on analyses of documents from an initial retrieval.
• Enriching document representations. Another obvious approach to bridge the gap between query and document terms is to enrich document representations. This strategy works well for short texts such as tweets [Efron et al., 2012] and noisy transcriptions of speech [Singhal and Pereira, 1999] . Although not as popular as query expansion techniques, researchers nevertheless explored this approach throughout the 1980s and 1990s Buckley, 1988b, Voorhees and Hou, 1993] . The origins of document expansion trace even earlier to Kwok [1975] , who took advantage of bibliographic metadata for expansion, and finally, Brauen et al. [1968] , who used previously issued user queries to modify the vector representation of a relevant document. Document expansion techniques have recently received renewed interest in the context of transformer architectures, as we will discuss in Section 3.5.
• Beyond exact term matching. There have been models that attempt to address the vocabulary mismatch problem without explicitly enriching query or document representations. A notable attempt is the statistical translation approach of Berger and Lafferty [1999] , who modeled retrieval as the translation of a document into a query in a noisy channel model. Their model attempted to learn translation probabilities between query and document terms, but these nevertheless represented mappings between discrete tokens.
In general, retrieval models up until this time stand in contrast to "soft" or semantic matching enabled by continuous representations in neural networks, where query terms do not have to match document terms exactly in order to contribute to relevance. Semantic matching refers to techniques and attempts to address a variety of linguistic phenomena, including synonymy, paraphrase, term variation, and different expressions of similar intents, specifically in the context of information access [Li and Xu, 2014] . Following this usage, "relevance matching" is often used to describes the correspondences between queries and texts that account for the text being relevant to the query (see Section 2.2). Thus, relevance matching is generally understood to comprise both exact match and semantic match components. However, there is another major phase in the development of ranking techniques before we get to semantic matching and how neural networks accomplish it.
BM25 and other term weighting schemes are typically characterized as unsupervised, although they contain free parameters (e.g., k 1 and b) that can be tuned given training data. The next major development in text ranking, beginning in the late 1980s is the application of supervised machinelearning techniques to learn ranking models: early examples include Fuhr [1989] , Wong et al. [1993] , and Gey [1994] . This approach, known as "learning to rank", 21 is typically based on hand-crafted, manually-engineered features, but these features remained primarily based on statistical properties of terms contained in the texts as well as intrinsic properties of the texts:
• The statistical properties of terms include functions of term frequencies, document frequencies, document lengths, etc., the same components that appear in a scoring function such as BM25.
In fact, BM25 scores between the query and various document fields (as well as scores based on other exact match scoring functions) are typically included as features in a learning to rank setup. Often, features incorporate proximity constraints, such as the frequency of a term pair co-occurring within five positions. Proximity constraints can be localized to a specific field in the text, for example, the co-occurrence of terms in the title of a web page or in anchor texts.
• Other intrinsic properties of the texts, ranging from very simple statistics such as the amount of JavaScript code on a web page or the ratio between HTML tags and content to more sophisticated measures such as the editorial quality or spam score as determined by a classifier. In the web context, features of the hyperlink graph, such as the count of inbound and outgoing links and PageRank scores, are common as well.
A real-world search engine can have hundreds of features (or even more). 22 This rise of learning to rank was driven largely by the growth in importance of search engines as indispensable tools for navigating the web, as earlier approaches based on human-curated directories (e.g., Yahoo!) became quickly untenable with the explosion of available content. Log data capturing behavioral traces of users (e.g., queries and clicks) could be used to improve machine-learned ranking models. A better search experience led to user growth, which yielded even more log data to further improve ranking quality-thus closing a self-reinforcing virtuous cycle (what Jeff Bezos calls "the flywheel"). A noteworthy innovation that played an important role in enabling this growth was a series of techniques for interpreting noisy user click data and converting them into training examples that could be fed into machine-learning algorithms [Joachims, 2002, Radlinski and Joachims, 2005] .
As we lack the space for a detailed treatment of learning to rank, we refer interested readers to two excellent surveys [Liu, 2009 , Li, 2011 and focus here on highlights that are most directly relevant for text ranking with transformers. At a high-level, learning-to-rank techniques can be divided into three basic type, based on the general form of their loss functions:
• A pointwise approach only considers losses on individual documents, transforming the ranking problem into classification or regression. 21 Note that "learning to rank" should not be understood as a phrase synonymous with "supervised machinelearning approaches to ranking". Rather, learning to rank refers to techniques that emerged during a specific period in history of information retrieval. Transformers for text ranking can be characterized as a supervised machine-learning approach, but would not generally be regarded as a learning to rank technique. Nevertheless, there are foundational concepts from learning to rank that are applicable to transformer-based approaches, which we identify throughout this survey. 22 https://googleblog.blogspot.com/2008/03/why-data-matters.html • A pairwise approach considers losses on pairs of documents, and thus focuses on preferences, that is, the property wherein A is more relevant (or preferred over) than B.
• A listwise approach considers losses on an entire list of documents, for example, directly optimizing a ranking metric such as normalized discounted cumulative gain (see Section 2.3 for a discussion of metrics).
Since this basic classification focuses on the form of the loss function, it is also appropriate to describe ranking techniques with transformers using the same terminology. Most techniques today can, in fact, be classified as pointwise approaches, and at least one can be described as adopting a pairwise approach. To date, we are not aware of any listwise approaches to text ranking using transformers.
Learning to rank reached its zenith in the early 2010s, on the eve of the deep learning revolution, with the development of models based on tree ensembles [Burges, 2010] . 23 At that time, there was an emerging consensus that gradient-boosted decision trees [Ganjisaffar et al., 2011] , in particular, represented the most effective solution to learning to rank. By that time, tree ensembles had been successfully deployed to solve a wide range of problems, perhaps most famously the Netflix Prize, which aimed to improve the quality of movie recommendations. 24
There is one key characteristic that distinguishes learning to rank from the deep learning approaches that came after. What's important is not the specific supervised machine-learning model: in fact, neural networks have been used since the early 1990s [Wong et al., 1993] , and RankNet, one of the most influential and well-known learning-to-rank models [Burges et al., 2005] , used a basic feedforward neural architecture. Instead, learning to rank is characterized by its use of discrete, usually hand-crafted features, numbering in the hundreds or even more. This contrast is explained in more detail next.
For text ranking, after learning to rank came deep learning, following initial excitement in the computer vision and then the natural language processing communities. In the context of information retrieval, deep learning approaches were exciting for two reasons: First, continuous vector representations freed text retrieval from the bounds of exact term matching (as already mentioned above, we'll see exactly how below). Second, neural networks promised to obviate the need for laboriously hand-crafted features (addressing a major difficulty with building systems using learning to rank).
In the space of deep learning approaches to text ranking, it makes sense to further distinguish "pre-BERT" models from BERT-based models (and more generally, transformer models). After all, the "BERT revolution" is the motivation for this survey to begin with. In the Deep Learning Track at TREC 2019, 25 the first large-scale evaluation of retrieval techniques following the introduction of BERT, its impact (more generally, the impact of pretrained neural language models) was clear from the effectiveness of the submissions . Analysis of the results showed that, taken as a coherent family of techniques, BERT-based models achieved substantially higher effectiveness than pre-BERT models, across implementations by different teams. The organizers of the evaluation recognized this as a meaningful distinction that separated two different "eras" in the development of deep neural approaches to text ranking.
This section provides a high-level overview of pre-BERT models. Needless to say, we do not have sufficient space to thoroughly detail roughly half a dozen years of model progression, and therefore refer the reader to existing surveys devoted to the topic [Onal et al., 2018 , Mitra and Craswell, 2019a . Note that here we focus specifically on models designed for document ranking and leave aside another vast body of literature, mostly from the NLP community, on the closely related problem of computing the semantic similarity between two sentences (for example, to detect if two sentences are paraphrases of each other). Models for these tasks share a number of architectural similarities, and indeed there has been cross-fertilization between the NLP and IR communities in this regard. However, there is one major difference: inputs to a model for computing semantic similarity are symmetric, i.e., Rel(s 1 , s 2 ) = Rel(s 2 , s 1 ), whereas queries and documents are 23 Although a specific thread of work in the learning to rank tradition, called "counterfactual learning to rank" [Agarwal et al., 2019] remains active to this day. 24 https://www.netflixprize.com/ 25 See Section 2.4 for an overview of what TREC is.
(b) a generic interaction-based neural ranking model Figure 1 : Two classes of pre-BERT neural ranking models. Representation-based models (left) independently learn vector representations of query and documents that can be compared to compute relevance scores using simple metrics such as cosine similarity. Interaction-based models (right) explicitly model term interactions in a similarity matrix that undergoes further processing to arrive at a relevance score.
obviously different and cannot be swapped as model inputs. The practical effect is that architectures for computing semantic similarity are symmetric, but may not be for modeling query-document relevance. Interestingly, though continuous dense representations for ranking are in some ways erasing the distinction between these two threads of work, as we will detail in Section 4.
Pre-BERT neural ranking models are generally classified into two categories: representation-based models and interaction-based models. Their high-level architectures are illustrated in Figure 1 . Representation-based models (left) focus on independently learning dense vector representations of queries and documents that can be compared to assess the relevance of the document via a simple metric such as cosine similarity or inner products. Interaction-based models (right) compare the representations of terms in the query with terms in the document directly to produce a similarity matrix that captures term interactions. This matrix then undergoes further processing to arrive at a query-document relevance score. In both cases, ranking models can incorporate a range of different neural components (e.g., convolutional neural networks and recurrent neural networks) to extract relevance signals from the similarity matrix.
Both representation-based and interaction-based models are usually trained end-to-end with relevance judgments, using only the embeddings of the query and document as input. Notably, additional features (hand-crafted or otherwise) are usually not used, which is a major departure from learning to rank. Below, we provide more details, with illustrative examples:
Representation-based models. This class of models (Figure 1 , left) learns vector representations of queries and documents, based on training data, that can be compared at ranking time to compute querydocument relevance scores. Since the query and document "arms" of the network are independent, this approach allows text representations to be computed offline. One of the earliest neural ranking models in the deep learning era, the Deep Structure Semantic Model (DSSM) proposed by Huang et al. [2013] constructs character n-grams from an input text (i.e., query or document) and passes the results to a series of fully-connected layers to produce a vector representation of the text. At retrieval time, query and document representations can then be compared with cosine similarity. Shen et al. [2014] improved upon DSSM by using CNNs to capture context. Rather than learning text representations as part of the model, the Dual Embedding Space Model (DESM) represents texts using pre-trained word2vec embeddings [Le and Mikolov, 2014] and computes relevance scores by aggregating cosine similarities across all query-document term pairs. Language models based on word embeddings [Ganguly et al., 2015] can be thought of as conceptually belonging to the class of representation-based models as well.
Interestingly, we are witnessing a resurgence of interest in representation-based approaches in the context of transformer architectures. The entirety of Section 4 is devoted to this topic.
Interaction-based models. This class of models (Figure 1 , right) explicitly captures "interactions" between terms from the query and terms from the document. These interactions are typically operationalized using a similarity matrix with rows corresponding to query terms and columns corresponding to document terms. Each entry m i,j in the matrix is typically populated with the cosine similarity between the embedding of the i-th query term and the embedding of the j-th document term. 26 At a high level, the operation of these models can be split into two steps: feature extraction and relevance scoring.
• In the feature extraction step, the model extracts relevance signals from the similarity matrix.
Conceptually, these represent different attempts to go beyond exact term matching using continuous vector representations of texts. 27 Unigram models like DRMM and KNRM [Xiong et al., 2017] aggregate the similarity between each query term and each document term, which can be viewed as histograms. DRMM creates explicit histograms, while KNRM uses Gaussian kernels to create differentiable "soft histograms" that allow the embeddings to be fine-tuned during training. Position-aware models like MatchPyramid [Pang et al., 2016] , PACRR [Hui et al., 2017] , Co-PACRR , and ConvKNRM [Dai et al., 2018] use CNNs to identify matches between sequences of query and document terms. MatchPyramid and PACRR accomplish this by consuming small regions of the similarity matrix using CNNs with various filter sizes. For example, a 2D CNN with a 3 × 3 filter can identify a sequence of three query term matches by identifying a diagonal in the similarity matrix. ConvKNRM, on the other hand, uses CNNs to combine sequences of terms into single embeddings before constructing similarity matrices. For example, a 1D CNN with a filter of size 3 takes three term embeddings as input and outputs a single embedding. ConvKNRM then creates one similarity matrix for each combination of filter sizes (e.g., comparing query unigrams to document unigrams, query bigram to document bigrams, etc.) before consuming each matrix analogously to KNRM.
• In the relevance scoring step, the features are combined and processed to produce a querydocument relevance score. This step often consists of applying pooling operations, concatenating the extracted features together, and then passing the resulting representation to a feedforward network that computes the relevance score.
While interaction-based models generally follow this high-level approach, many variants have been proposed that incorporate additional components. For example, POSIT-DRMM [McDonald et al., 2018] uses an LSTM to contextualize static embeddings before comparing them. EDRM [Liu et al., 2018b] extends ConvKNRM by incorporating entity embeddings, which are treated like an additional filter size in ConvKNRM (e.g., query entity embeddings are compared with document unigram term embeddings). HiNT [Fan et al., 2018b] splits the document into passages, creates one similarity matrix for each, and combines the passage-level signals to predict a single document-level relevance score. The NPRF framework incorporates feedback documents by using a neural ranking method, like KNRM, to predict their similarity to a target document being ranked.
In general, studies have shown interaction-based models to be more effective but slower than representation-based models. The latter reduces text ranking to simple similarity comparisons between query vectors and precomputed document vectors, which can be accelerated by techniques for approximate nearest neighbor search (see Section 4.2). In contrast, interaction-based models are typically deployed as rerankers over a candidate set of results retrieved by an exact match technique (see Section 3.2 and Section 3.4). Hybrid models. Finally, representation-based and interaction-based approaches are not mutually exclusive. A well-known and early hybrid is the DUET model [Mitra et al., 2017, Mitra and Craswell, 2019b] , which augments a representation-learning component with an interaction-based component responsible for identifying exact query term matches.
There has undeniably been significant research activity throughout the 2010s exploring a wide range of neural architectures for document ranking, but how far has the field concretely advanced, particularly since approaches based on deep learning require large amounts of training data? posed the provocative question, asking if neural ranking models were actually better than "traditional" keyword-matching techniques in the absence of vast quantities of training data available from behavior logs (i.e., queries and clickthroughs). This is an important issue because academic researchers have faced a perennial challenge in obtaining access to such data, which are available to only researchers in industry (with rare exceptions). To what extent do neural ranking models "work" on the limited amounts of training data that are publicly available? answered this question by comparing a number of prominent interaction-based and representation-based neural ranking models to a well-engineered implementation of bag-ofwords querying with well-tuned query expansion on the dataset from the TREC 2004 Robust Track [Voorhees, 2004] . Under this limited data condition, most of the neural ranking methods were unable to beat the keyword-querying baseline. reproduced the same finding for an expanded pool of neural ranking methods with completely different implementations, thus increasing the veracity of the original findings. Of course, many of the papers cited above report significant improvements when trained with large, proprietary datasets, which is certainly believable, but the results are difficult to validate.
With BERT, though, everything changed, nearly overnight.
BERT arrived on the scene in October 2018. The first application of BERT to text ranking was reported by Nogueira and Cho [2019] in January 2019 on the MS MARCO passage retrieval test collection [Bajaj et al., 2018] , where the task is to rank passages (paragraph-length extracts) from web pages with respect to users' natural language queries from Bing query logs (see more details in Section 2.5). The relevant portion of the leaderboard at the time is presented in Table 1 , showing Microsoft's BM25 baselines and the effectiveness of IRNet, the best system right before the introduction of BERT (see Section 2.3 for the exact definition of the metric). Within less than a week, effectiveness shot up by near seven points 28 absolute, which is a ∼30% relative gain. It is possible to explain these gains in an intuitively interpretable manner: based on the definition of the metric, an MRR of 0.281 implies that, on average, the first relevant result appears in rank position 3.6; an MRR of 0.359 implies that, on average, the first relevant result appears in rank position 2.8. On a mobile phone with limited screen real estate, this difference might translate into whether or not the user must scroll to see the relevant answer.
Such a big jump in effectiveness that can be directly attributed to a single innovation, especially with a simple adaptation of an existing model, is rarely seen in either academia or industry, which led to immediate excitement in the community. The simplicity of the model led to rapid widespread replication of the results. The skepticism expressed by was retracted in short order , as many researchers quickly demonstrated that with pretrained transformer models, large amounts of relevance judgments were not necessary to build effective models for text ranking. The availability of the MS MARCO passage retrieval test collection further mitigated data availability issues. The combination of these various factors meant that, nearly overnight, exploration at the forefront of neural models for text ranking was within reach of academic research groups, and was not limited to researchers in industry who had the luxuries of access to query logs. Nogueira and Cho [2019] kicked off the "BERT revolution" for text ranking, and the research community quickly set forth to build on their results-addressing limitations and expanding the work in various ways. Looking at the leaderboard today, the dominance of BERT remains evident, just by looking at the names of the submissions. The rest, as they say, is history. The remainder of this survey is about that history.
The target audience for this survey is a first-year graduate student or perhaps an advanced undergraduate. As this is not intended to be a general introduction to natural language processing or information retrieval, we assume that the reader has basic background in both. For example, we discuss sequence-to-sequence formulations of text processing problems (to take an example from NLP) and query evaluation with inverted indexes (to take an example from IR) assuming that the reader has already encountered these concepts before.
Furthermore, we expect that the reader is already familiar with neural networks and deep learning, particularly pre-BERT models (for example, CNNs and RNNs). Although we do provide an overview of BERT and transformer architectures, that material is not designed to be tutorial in nature, but merely intended to provide the setup of how to apply transformers to text ranking problems.
This survey is organized as follows:
• Setting the Stage (Section 2). We begin with a more precise characterization of the problem we are tackling in the specific context of information retrieval. This requires an overview of modern evaluation methodology, involving discussions about information needs, notions of relevance, ranking metrics, and the construction of test collections. By no means is this intended to be an introductory tutorial: our coverage extends only to those concepts that are critical to understanding the results presented in the rest of the survey. • Transformers in Multi-Stage Ranking Architectures (Section 3). The most straightforward application of transformers to text ranking is as reranking models to improve the output of candidates generated by keyword search. This section details various ways this basic idea can be realized in the context of multi-stage ranking architectures. • Continuous Dense Representations for Ranking (Section 4). Recent approaches based on representational learning have tried to reformulate text ranking into simple and efficient comparisons between continuous dense vectors representations of text. This section covers efforts to replace sparse bag-of-words representations central to candidate generation in multi-stage ranking architectures, which pave the way for novel ranking architectures. • Future Directions and Conclusions (Section 5). We have only begun to scratch the surface in applications of transformers to text ranking. This survey concludes with discussions of interesting open problems and our attempts to prognosticate where the field is heading.
Given limits in both time and space, it is impossible to achieve comprehensive coverage, even in a narrow circumscribed topic, both due to the speed at which research is progressing and the wealth of connections to closely related topics.
This survey focuses on what might be characterized as "core" text ranking. Noteworthy intentional omissions include other aspects of information access such as question answering, summarization, and recommendation, despite their close relationship to the material we cover. Each of these topics would occupy an equally lengthy survey for an adequate treatment. Our focus on "core" text ranking means that we do not elaborate on how ranked results may be used to directly supply answer spans (as in typical formulations of question answering), how multiple results may be synthesized (as in summarization), and how systems may suggest related texts based on more than just content (as in recommendations).
This section begins by more formally characterizing the text ranking problem, explicitly enumerating our assumptions about characteristics of the input and output, and more precisely circumscribing the scope of this survey. In this exposition, we will adopt the perspective of information access, focusing specifically on the problem of ranking texts with respect to their relevance for a particular query-what we have characterized as the "core" text ranking problem. However, most of our definitions and discussions carry straightforwardly to other ranking objectives, such as the diverse applications discussed in Section 1.1.
From the evaluation perspective, this survey focuses on what is commonly known as the Cranfield paradigm, an approach to systems-oriented evaluation of information retrieval (IR) systems based on a series of experiments by Cyril Cleverdon and his colleagues in the 1960s. For the interested reader, Harman [2011] provides a nice overview of the early history of IR evaluation. Also know as "batch evaluations", the Cranfield paradigm has come to dominate the IR research landscape over the last half a century. Nevertheless, there are other evaluation paradigms worth noting: interactive evaluations place humans "in the loop" and are necessary to understand the important role of user behavior in information seeking [Kelly, 2009] . Online services with substantial numbers of users can engage in experimentation using an approach known as A/B testing [Kohavi et al., 2007] . Despite our focus on the Cranfield paradigm, primarily due to its accessibility to the intended audience of our survey, evaluations from multiple perspectives are necessary to accurately characterize the effectiveness of a particular technique.
The formulation of text ranking assumes the existence of a collection or corpus C = {d i } comprised of mostly unstructured natural language text. We say "mostly unstructured" because texts are, of course, typically broken into paragraphs, with section headings and other discourse markers-these can be considered a form of "structure". This stands in contrast to, for example, tabular data or semi-structured logs, which are textual in nature as well. We specifically consider such types of textual data out of scope in this survey.
Our collection C can be arbitrarily large (but finite)-in the case of the web, countless billions of pages. This means that issues related to computational efficiency, for example the latency and throughput of text ranking, are important considerations, especially in production systems. We mostly set aside issues related to multilinguality and focus on English, although there are straightforward extensions to much of the material discussed in this survey to other languages.
It is further assumed that the corpus is provided "ahead of time" to the system, prior to the arrival of queries, and that a "reasonable" amount of offline processing may be conducted on the corpus. This constraint implies that the corpus is mostly static, in the sense that additions, deletions, or modifications to texts happen in batch or at a pace that is slow compared to the amount of preprocessing required by the system for proper operation. 29 This assumption becomes important in the context of the document preprocessing techniques we discuss in Section 3.5.
Texts can vary in length, ranging from sentences (e.g., searching for related questions in a community question answering application) to entire books, although the organization of the source texts, how they are processed, and the final granularity of ranking can be independent. To illustrate: in a collection of full-text scientific articles, we might choose to only search the article titles and abstracts. An alternative might be to segment full-text articles into paragraphs and consider each paragraph as the unit of retrieval, i.e., the system returns a list of paragraphs as results. Yet another alternative might be to rank articles by aggregating evidence across paragraphs-that is, the system treats paragraphs as the atomic unit of analysis, but for the goal of producing a ranking of the articles from which those paragraphs are drawn. As we shall see, the issue of text length is an important consideration in applications of transformer architectures to text ranking. In many cases, the texts are longer than the maximum allowable sequence length that current transformer-based models can accept, necessitating additional techniques to address this limitation. Figure 2 : Example "topic" for ad hoc retrieval from the TREC 2004 Robust Track, comprised of "title", "description", and "narrative".
Having sufficiently characterized the corpus, we now turn our attention to queries. In the web context, short keyword queries that a user types into a search box are merely the external manifestations of an information need, which is the motivation that compelled the user to seek information in the first place. Belkin [1980] calls this an "anomalous state of knowledge" (ASK), where searchers perceive gaps in their cognitive states with respect to some task or problem; see also Belkin et al. [1982a,b] . Strictly speaking, queries are not synonymous with information needs [Taylor, 1962] . The same information need might give rise to different manifestations with different systems: for example, a few keywords are typed into the search box of a web search engine, but a fluent, well-formed natural language question is spoken to a voice assistant. 30
In this survey, we are not concerned with the cognitive processes underlying information seeking, and focus on the workings of text ranking models after they have received a tangible signal to process. Thus, we somewhat abuse the terminology and refer to "the thing" that the ranking is computed with respect to as the query, and use it as a metonym for the underlying information need. In other words, although the query is not the same as the information need, we really only care about what is fed to the ranking model, in which case this distinction is not particularly important. We only consider queries that are expressed in text, although in principle queries can be presented in different modalities, for example, speech 31 or images, or even "query by humming" [Ghias et al., 1995] .
Nevertheless, to enable automated processing, information needs must be encoded in some representation. In the Text Retrieval Conferences (TRECs), an influential series of community evaluations in information retrieval (see Section 2.4), information needs are operationalized as "topics". 32 Figure 2 provides an example from the TREC 2004 Robust Track.
A TREC topic for ad hoc retrieval is comprised of three fields:
• the "title", which comprises a few keywords that describe the information need, close to a query that a user would type into a search engine;
• the "description", typically a well-formed natural language sentence that describes the desired information; and,
• the "narrative", a paragraph of prose that details the characteristics of the desired information, particularly nuances that are not articulated in the title or description. 30 In the latter case, researchers might refer to these as voice queries, but it is clear that spoken utterances are very different from typed queries, even if the underlying information need is the same. 31 Spoken queries can be transcribed into text with the aid of automatic speech recognition (ASR) systems. 32 Even within TREC, topic formats have evolved over time, but the structure we describe has been stable since TREC-7 in 1998 [Voorhees and Harman, 1998 ].
In most information retrieval evaluations, the title is the query that is fed into a system for evaluation purposes. Some papers explicitly state, "title queries" or something to that effect, but many papers omit this detail, in which case it is usually safe to assume that the title were used as queries.
Although in actuality the narrative is a more faithful description of the information need, i.e., what the user really wants, in most cases feeding the narrative into a ranking model leads to poor results because the narrative often contains terms that are not important to the topic. These extraneous terms serve as distractors to a ranking model based on exact term matches, since such a model will try to match all query terms. 33 Although results vary by domain and the specific set of information needs used for evaluation, one common finding is that either the title or the title and description concatentated together yields the best results; see, for example, Walker et al. [1997] . However, the differences in effectiveness between the two conditions are usually small. Nevertheless, the key takeaway here is that the expression of the information need that is fed to a ranking model often has a substantive effect on retrieval effectiveness. We will see that this is particularly the case for BERT (see Section 3.3.2).
Having more precisely described the inputs, we can now formally define the text ranking problem:
Given an information need expressed as a query q, the text ranking task is to return a ranked list of k texts {d 1 , d 2 . . . d k } from an arbitrarily large but finite collection of texts C = {d i } that maximizes a metric of interest, for example, nDCG, AP, etc.
Descriptions of a few common metrics are provided in Section 2.3, but they all aim to quantify the "goodness" of the results with respect to the information need. The ranking task is also called top k retrieval, where k is the length of the ranked list (also known as the retrieval depth).
The "thing" that performs the ranking is referred to using different terms in the literature: {ranking, retrieval} model, {ranking, retrieval, scoring} function, or even just "the system" when discussed in an end-to-end context. In this survey, we tend to use the term "ranking model", but consider all these terms roughly interchangeable. Typically, the ranked texts are associated with scores, and thus the output of a ranking model can be more explicitly characterized as
There is one final concept necessary to connect the query, as an expression of the information need, to the "goodness" of the ranked texts according to some metric: Ultimately, the foundation of all ranking metrics rests on relevance, 35 which is a relation between a text and a particular information need. A text is said to be relevant if it addresses the information need, otherwise it is not relevant. However, this binary treatment of relevance is itself a simplification, as it is more accurate to characterize relevance using ordinal scales in multiple dimensions [Spink and Greisdorf, 2001] . Discussions and debates about the nature of relevance are almost as old as quest for building automated search systems themselves (see Section 1.2), since relevance figures into discussions of what such systems should return and how to evaluate the quality of their output. Countless pages have been written about relevance, from different perspectives ranging from operational considerations (i.e., for designing search systems) to purely cognitive and psychological studies (i.e., how humans assimilate and use information acquired from systems). Saracevic [2017] offered a relatively recent survey after having studied the problem for at least half a century [Saracevic, 1975] .
While seemingly intuitive, relevance is surprisingly difficult to precisely define. Furthermore, the information science literature discusses many types of relevance; for the purposes of measuring search quality, information retrieval researchers are generally concerned with topical relevance, or the "aboutness" of the document-does the topic or subject of the text match the information need? There are other possible considerations as well: for example, cognitive relevance, e.g., whether the text is understandable by the user, or situational relevance, e.g., whether the text is useful for solving the problem at hand. To illustrate these nuances: A text might be topically relevant, but is written for experts whereas the searcher desires an introductory text; thus, it is not relevant from the cognitive 33 Prior to the advent of neural networks, researchers have attempted to extract "key terms" or "key phrases" from so-called "verbose" queries, e.g., Bendersky and Croft [2008] , though these usually refer to sentence-length descriptions of information needs as opposed to paragraph-length narratives. 34 A minor complication is that ranking models might produce scoring ties, which need to be resolved at evaluation time since many metrics assume monotonically increasing ranks; see Section 2.3 for more discussion. 35 "Relevancy" is sometimes used, often by industry practitioners. However, information retrieval researchers use the term "relevance". perspective. A text might be topically relevant, but the user is searching for information to aid in making a specific decision-for example, whether to send a child to public or private school-and while the document provides helpful background information, it provides no actionable advice. In this case, we might say that the document is topically relevant but not useful, i.e., from the perspective of situational relevance. Although it has been well understood for decades that relevance is a complex phenomenon, there remains a wide gap between studies that examine these nuances and the design of search systems and ranking models, as it is not clear how such insights can be operationalized.
More to the task at hand: in terms of developing ranking models, the most important lesson from many decades of information retrieval research is that relevance is in the eye of the beholder, that it is a user-specific judgment about a text that involves complex cognitive processes. To put more simply: for my information need, I am the ultimate arbiter of what's relevant or not, nobody else's opinion counts or matters. Thus, relevance judgments represent a specific person's assessment of what's relevant or not-this person is called the assessor (or sometimes the annotator). In short, all relevance judgements are opinions. Relevance is not a "truth" (in a platonic sense) or an "inherent property" of a piece of text that the assessor attempts to "unlock".
In this way, relevance differs quite a bit from human annotations in NLP applications, where (arguably), there is, for example, the true part of speech tag of a word or dependency relation between two words. Trained annotators can agree on a word's part of speech nearly all the time, and disagreements are interpreted as the result of a failure to properly define the subject of annotation (i.e., what a part of speech is). It would be odd to speak of an annotator's opinion of a word's part of speech, but that is exactly what relevance is: an assessor's opinion concerning the relation between a text and an information need.
With this understanding, it shouldn't be a surprise then that assessor agreement on relevance judgments is quite low: 60% overlap is a commonly cited figure [Voorhees, 2000] , but the range of values reported in the literature vary quite a bit (from around 30% to greater than 70%), depending on the study design, the information needs, and the exact agreement metric; see [Harman, 2011] for a discussion of this issue across studies spanning many decades. The important takeaway message is that assessor agreement is far lower than values an NLP researcher would be comfortable with for a human annotation task (κ > 0.9 is sometimes used as a reference point for what "good" agreement means). The reaction from an NLP researcher would be, "we need better annotation guidelines". This, however, is fundamentally not possible, as we explain below.
Why is agreement so low among relevance judgments provided by different assessors? First, it is important to understand the setup of such experiments. Ultimately, all information needs arise from a single individual. In TREC, a human assessor writes the topic, which represents a best effort articulation of the information need relatively early in the information seeking process. Topics are formulated after some initial exploratory searches, but before in-depth perusal of texts from the corpus. The topics are then released to teams participating in the evaluation, and the same individual who wrote the topic then assesses system outputs (see Section 2.4 for more details).
Thus, if we ask another assessor to produce an independent set of relevance judgments (for example, in the same way we ask multiple annotators to produce part-of-speech tags for a corpus in an NLP setting in order to compute inter-annotator agreement), such a task is based on a particular representation of that information need (e.g., a TREC topic, as in Figure 2 ). 36 Thus, the second individual is judging relevance with respect to an interpretation of that representation. Remember, the actual characteristics of the desired information is a cognitive state that lies in the user's head, i.e., Belkin's anomalous state of knowledge. Furthermore, in some cases, the topic statements aren't even faithful representations of the true information need to begin with: details may be missing and inconsistencies may be present in the representations themselves. The paradox of relevance is that if a user were able to fully and exhaustively articulate the parameters of relevance, there may likely be no need to search in the first place-for the user would already know the information desired.
We can illustrate with a concrete example based on the TREC topic shown in Figure 2 about "black bears attacks": consider, would documents about brown (grizzly) bears be relevant? 37 It could be the case that the user is actually interested in attacks by bears (in general), and just happens to have referenced black bears as a starting point. It could also be the case that the user specifically wants only attacks by black bears, perhaps to contrast with the behavior of brown bears. Or, it could be the case that the user isn't familiar with the distinction, started off by referencing black bears, and only during the process of reading initial results is a decision made about different types of bears. All three scenarios are plausible based on the topic statement, and it can be seen now how different interpretations might give rise to very different judgments.
Beyond these fundamental issues, which center around representational deficiencies of cognitive states, there are issues related to human performance. Humans forget how they interpreted a previously encountered text and may judge two similar texts inconsistently. There may be learning effects that carry across multiple texts: for example, one text uses terminology that the assessor does not recognize as being relevant until a second text is encountered that explains the terminology. In this case, the presentation order of the texts matters, and the assessor may or may not reexamine previous texts to adjust the judgments. There are also more mundane factors: Assessors may get tired and misread the material presented. Sometimes, they just make mistakes (e.g., clicked on the wrong button in an assessment interface). All of these factors further contribute to low agreement.
One obvious question that arises from this discussion is: With such low inter-annotator agreement, how are information retrieval researchers able to reliably evaluate systems at all? Given the critical role that evaluation methodology plays in any empirical discipline, it should come as no surprise that researchers have examined this issue in detail. In studies where we have multiple sets of relevance judgments (i.e., from different assessors), it is easy to verify that the score of a system does indeed vary (often, quite a bit) depending on which set the system is evaluated with (i.e., whose opinion of relevance). However, the ranking of a group of systems is stable with respect to assessor variations [Voorhees, 2000] . 38 How stable? Exact values depend on the setting, but measured in terms of Kendall's τ , a standard rank correlation metric, values consistently > 0.9 are observed. That is, if system A is better than system B, the score of system A will likely be higher than the score of system B, regardless of the relevance judgments used for evaluation. 39 This is a widely reproduced and robust finding, and these conclusions have been shown to hold across many different retrieval settings [Sormunen, 2002 , Trotman and Jenkinson, 2007 , Bailey et al., 2008 .
This means that while the absolute value of an evaluation metric must be interpreted cautiously, comparisons between systems are generally reliable given a well-constructed test collection; see more discussions in Section 2.4. The inability to quantify system effectiveness in absolute terms is not a limitation outside of the ability to make marketing claims, as most research is focused on the effectiveness of a particular proposed innovation, and so the desired comparison is a ranking model with and without that innovation.
Ranking metrics quantify the quality of a ranking of texts and are computed from relevance judgments, also called qrels. In order to be consistent with the information retrieval literature, here we focus on ranking documents. Formally, relevance judgments comprise a set of (q, d, r) triples, where the relevance judgment r is a (human-provided) annotation on (q, d) pairs. In the simplest case, r is a binary variable-either document d is relevant to query q, or it is not relevant. A three-way scale of not relevant, relevant, and highly-relevant is one common alternative, and in web search, a five-point scale is often used-perfect, excellent, good, fair, and bad-which even has an acronym PEGFB. 40 Non-binary relevance judgments are called a graded relevance judgments: "graded" is used in the sense of "grade", defined as "a position in a scale of ranks or qualities" (from the Merriam-Webster Dictionary). 38 Note that while studies of assessor agreement predate this paper by several decades at least, for example, Lesk and Salton [1968] , the work by Voorhees is generally acknowledged as establishing these findings in the context of modern test collections. 39 Conflated with this high-level summary is the effect size, i.e., the "true" difference between the effectiveness of systems, or an inferred estimate thereof. With small effect sizes, system A vs. system B comparisons are less likely to be consistent across different assessors. Not surprisingly, [Voorhees, 2000 ] studied this as well; see for a more recent examination in a different context. 40 Yes, there are those who actually try to pronounce this utterly unpronounceable jumble of letters.
Practically speaking, relevance judgments are text files that can be downloaded as part of a test collection (see Section 2.4) and can be treated like "ground truth" in an NLP task. 41 The ranked lists produced by a system (using a particular approach) for a set of queries (in TREC, topics) is called a "run". The qrels and the run file are fed into an evaluation program such as trec_eval, the most commonly used program by information retrieval researchers, which automatically computes a litany of metrics. These metrics define the hill to climb in the quest for effectiveness improvements.
Below, we describe a number of common metrics that are used throughout this survey. To be consistent with the literature, we largely follow the notation and convention of Mitra and Craswell [2019a] . We rewrite a ranked list
, retaining only the rank i induced by the score s i 's. Many metrics are computed at a particular cutoff (or have variants that do so), which means that the ranked list R is truncated to a particular length k,
, where k ≤ l: this is notated as Metric@k. The primary difference between l and k is that the system decides l (i.e., how many results to return), where as k is a defined by the evaluation metric (e.g., by the organizers of an evaluation or the authors of a paper). Sometimes, l and k are left unspecified, in which case it is usually the case that l = k = 1000. In most TREC evaluations, runs contain up to 1000 results per topic, and the metrics evaluate the entirety of the ranked lists (unless an explicit cutoff is specified).
From a ranked list R, we can compute the following metrics:
Precision is defined as the fraction of documents in ranked list R that are relevant, or:
where rel(q, d) indicates whether document d is relevant to query q, assuming binary relevance. Graded relevance judgments are binarized with some relevance threshold, e.g., in a three-grade scale, we might set rel(q, d) = 1 for "relevant" and "highly relevant" judgments. Often, precision is evaluated at a cutoff k, notated as Precision@k or abbreviated as P@k. If the cutoff is defined in terms of the number of relevant documents for a particular topic (i.e., a topic-specific cutoff), the metric is known as R-precision.
This metric has the advantage that it is easy to interpret: of the top k results, what fraction are relevant? 42 There are two main downsides: First, precision does not take into account graded relevance judgments, and for example, cannot separate "relevant" from "highly relevant" results since the distinction is erased in rel(q, d). Second, precision does not take into account rank positions (beyond the cutoff k). For example, consider P@10: relevant documents appearing at ranks one and two (with no other relevant documents) would receive a precision of 0.2; P@10 would be exactly the same if those two relevant documents appeared at ranks 9 and 10. Yet, clearly, the first ranked list would be preferred by a user.
Recall is defined as the fraction of relevant documents (in the entire collection C) for q that are retrieved in ranked list R, or:
where rel(q, d) indicates whether document d is relevant to query q, assuming binary relevance. Graded relevance judgments are binarized in the same manner as precision.
Mirroring precision, recall is often evaluated at a cutoff k, notated as Recall@k or abbreviated R@k. This metric has the same advantages and disadvantages as precision: it is easy to interpret, but does not take into account relevance grades or the rank positions in which relevant documents appear. 43 41 However, IR researchers tend to avoid the term "ground truth" because relevance judgments are opinions; see Section 2.2. 42 There is a corner case here if l < k: for example, what is P@10 for a ranked list that only has five results? One possibility is to always use k in the denominator, in which case the maximum possible score is 0.5; this has the downside of averaging per topic scores that have different ranges when summarizing effectiveness across a set of topics. The alternative is to use l as the denominator. Unfortunately, treatment is inconsistent in the literature. 43 Note that since the denominator in the recall equation is the total number of relevant documents, the symmetric situation of what happens when l < k does not exist as it does with precision. However, a different issue
Reciprocal rank (RR) is defined as:
where rank i is the smallest rank number of a relevant document. That is, if a relevant document appears in the first position, reciprocal rank = 1, 1/2 if it appears in the second position, 1/3 if it appears in the third position, etc. Like precision and recall, RR is computed with respect to binary judgments. Although RR has an intuitive interpretation, it only captures the appearance of the first relevant result. For question answering or tasks in which the user may be satisfied with a single answer, this may be an appropriate metric, but reciprocal rank is usually a poor choice for ad hoc retrieval because users usually desire more than one relevant document. As with precision and recall, reciprocal rank can be computed at a particular rank cutoff, denoted with the same @k convention.
Average Precision (AP) is defined as:
where all notation used have already been defined. The intuitive way to understand average precision is that it is the average of precision scores at cutoffs corresponding to the appearance of every relevant document; rel(q, d) can be understood as a binary indicator variable, where non-relevant documents contribute nothing. Relevant documents that don't appear in the ranked list at all contribute zero to the average; note that the denominator is the total number of relevant documents. Once again, relevance is assumed to be binary.
Typically, average precision is measured without a cutoff, i.e., at rank 1000, the default length of l used in most evaluations. This is because the metric factors in retrieval of all relevant documents, and thus a cutoff would artificially reduce the score (i.e., it has the effect of including a bunch of zeros in the average for relevant documents that do not appear in the ranked list). Evaluations use average precision when the task requires measurement of recall, so imposing a cutoff that is too small makes little sense. The default cutoff of 1000 is a compromise between accurate measurement and practicality: in practice, relevant documents appearing below rank 1000 contribute negligibly to the final scores (which are usually reported to four digits after the decimal point), and run submissions with 1000 hits per topic are still manageable in size.
Average precision is more difficult to interpret, but it is a single summary statistic that captures aspects of both precision and recall, while favoring appearance of relevant documents towards the top of the ranked list.
Normalized Discounted Cumulative Gain (nDCG) is a metric that is most frequently used to measure the quality of web search results. Unlike the other metrics above, nDCG was specifically designed for graded relevance judgments. For example, if relevance were measured on a five-point scale, rel(q, d) would return r ∈ {0, 1, 2, 3, 4}. First, we define Discounted Cumulative Gain (DCG):
Gain is used here in the sense of utility, i.e., how much value does a user derive from a particular result. There are two factors that go into this calculation: the relevance grade (i.e., highly relevant results are "worth" more than relevant results) and the rank at which the result appears (relevant results near the top of the ranked list are "worth" more). The discounting refers to the decay in the gain (utility) as the user consumes results lower and lower in the ranked list. Finally, we introduce normalization:
emerges when k is smaller than the total number of relevant documents, in which case perfect recall is not possible. Therefore, it is inadvisable to set k to a value smaller than the total number of relevant documents across all topics in a test collection. While in most cases, k is fixed for all topics in a test collection, there are variants where k varies per topic, for example, as a function of the number of (known) relevant documents for that topic.
where IDCG represents the DCG of an "ideal" ranked list: this would be a ranked list that begins with all of the documents of the highest relevance grade, then the documents with the next highest relevance grade, etc. Thus, nDCG represents DCG normalized to a range of [0, 1] with respect to the best possible ranked list. Typically, nDCG is associated with a rank cutoff; a value of 10 or 20 is common. Since most commercial web search engines present ten results on a page (on the desktop, at least), these two settings represent nDCG with respect to the first or first two pages of results. For similar reasons, nDCG@3 or nDCG@5 are often used in the context of mobile search, given the much smaller screen sizes of phones.
This metric is popular for evaluating the results of web search for a number of reasons: First, nDCG can take advantage of graded relevance judgments, which provide finer distinctions on output quality. Second, the discounting and cutoff presents a reasonably accurate (albeit simplified) model of realworld user behavior, as revealed through eye-tracking studies; see, for example, Joachims et al. [2007] . Users do tend to scan results linearly, with increasing probability of "giving up" and "losing interest" as they consume more and more results (i.e., proceed further down the ranked list). This is modeled in the discounting, and there are variants of nDCG that apply different discounting schemes to model this aspect of user behavior. The cutoff value models a hard stop when users stop reading. For example, nDCG@10 quantifies the result quality of the first page of search results in a browser, assuming the user never clicks "next page" (which is frequently the case).
All of the metrics we have discussed above quantify the quality of a single ranked list with respect to a specific topic. Typically, the arithmetic mean across all topics in a test collection is used as a single summary statistic to denote the quality of a run for those topics. 44 We emphasize that it is entirely meaningless to compare effectiveness scores from different test collections (since scores do not control for differences due to corpora, topic difficulty, and many other issues), and even comparing a run that participated in a particular evaluation with a run that did not can be fraught with challenges (see next section).
A few additional words of caution: this aggregation can hide potentially big differences in per-topic scores. Some topics are "easy" and some topics are "difficult", and it is certainly possible that a particular ranking model has an affinity towards certain types of information needs. These nuance are all lost in a simple arithmetic mean across per-topic scores.
There is one frequently unwritten detail that is critical to the interpretation of metrics worth discussing. What happens if the ranked list R contains a document for which no relevance judgement exists, i.e., the document does not appear in the qrels file for that topic? This is called an "unjudged document", and the standard treatment (by most evaluation programs) is to consider unjudged documents not relevant. Unjudged documents are quite common because it is impractical to exhaustively assess the relevance of every document in a collection with respect to every information need; the question of how to select documents for assessment is discussed in the next section, but for now let's just take this observation as a given.
The issue of unjudged documents is important because of the assumption that unjudged are not relevant. Thus, a run may be poor not because the ranking model is poor, but because the ranking model produces many results that are unjudged (again, assume this as a given for now, we discuss why this may be the case in the next section). The simplest way to diagnose potential issues to is compute the fraction of judged documents at cutoff k (Judged@k or J@k), for example, if we find that 80% of the results in the top 10 hits are unjudged, Precision@10 is capped at 0.2. There is no easy fix to this issue beyond diagnosing and noting it: assuming that unjudged documents are not relevant is perhaps too pessimistic, but the opposite and only other reasonable alternative of assuming that unjudged documents are relevant is equally suspect.
There is a final detail worth explicitly mentioning. All of the above metrics assume that document relevance scores are strictly decreasing, and that there are no score ties. Otherwise, the evaluation program must arbitrarily make some decision to map identical scores to different ranks (necessary because metrics are defined in terms of rank order). For example, trec_eval breaks ties based on the reversed lexicographical ordering of the document ids. These arbitrary decisions introduce potential differences across alternative implementations of the same metric. Most recently, quantified the effects of scoring ties from the perspective of experimental repeatability and noted that the impact of score ties can be metric differences up to the third place after the decimal point. While the overall effects are small and not statistically significant, to eliminate this experimental confound, they advocated that a system ensures that there are no scoring ties in its ranked output-rather than let the evaluation program make arbitrary decisions. 45 Of course, Lin and Yang were not the first to examine this issue, see for example, Cabanac et al. [2010] , Ferro and Silvello [2015] for additional discussions.
We conclude this section with a number of remarks, some of which represent conventions and tacit knowledge by the community that are rarely explicitly communicated:
• Naming metrics. Mean average precision, abbreviated MAP, represents the mean of average precision scores across many topics. Similarly, mean reciprocal rank, abbreviated MRR, represents the mean of reciprocal rank scores across topics. 46 In some papers, the phrase "early-precision" is used to refer to the quality of top ranked results-as measured by a metric such as Precision@k or nDCG@k with a relatively small cutoff (e.g., k = 10). It is entirely possible for a system to excel at early precision (i.e., identify a few relevant documents and place them near the top of the ranked list) but not necessarily be effective when measured using recall-oriented metrics (which requires identifying all relevant documents). • Reporting metrics. Most test collections or evaluations adopt an official metric, or sometimes, a few official metrics. It is customary when reporting results to at least include those official metrics. 47 The choice of metric is usually justified by the creators of the test collection or the organizers of the evaluation (e.g., we aim to solve this problem, and the quality of the solution is best captured by this particular metric). Unless there is a compelling reason otherwise, follow established conventions; otherwise, results will not be comparable. It has been a convention, for example, at TREC, that metrics are reported to four places after the decimal, e.g., 0.2932. In prose, a unit of 0.01 in score is often referred to as a point, as in, an improvement from 0.19 to 0.29 is a ten-point gain. In some cases, metrics are reported in these terms, e.g., multiplied by 100, so 0.2932 becomes 29.32. 48 Finally, recognizing that a difference of 0.001 is just noise, some researchers opt to only report values to three digits after the decimal point, so 0.2932 becomes 0.293. • Comparing metrics. Entire tombs have been written about proper evaluation practices when comparing results, starting with statistical tests of significance. As we lack the space for a detailed exposition, we refer the reader to Sakai [2014] and Fuhr [2017] as starting points into the literature.
Having defined metrics for measuring the quality of a ranked list, we have now described all components of the text ranking problem: Given an information need expressed as a query q, the text ranking task is to return a ranked list of k texts {d 1 , d 2 . . . d k } from an arbitrarily large but finite collection of texts C = {d i } that maximizes a metric of interest. Where are the resources we need to concretely tackle this challenge? We turn our attention to this next.
Based on the discussions above, we can enumerate the ingredients necessary to evaluate a text ranking algorithm: a corpus or collection of texts to search, a set of information needs (i.e., topics), relevance judgments (or qrels) for those needs. Together, these comprise the components of what is known as a test collection for information retrieval research. With a test collection, it become straightforward to generate rankings with a particular ranking model and then compute metrics to quantify the quality 45 This was accomplished by first defining a consistent tie-breaking procedure and then subtracting a small to the tied scores to induce the updated rank ordering. 46 Some texts use MAP to refer to the score of a specific topic, which is technically incorrect. This is related to a somewhat frivolous argument on metric names that has raged on in the information retrieval community for decades now: there are those who argue that even the summary statistic across multiple topics for AP should be referred to as AP. They point as evidence the fact that we never write "MP@5" (i.e., mean precision at rank cutoff 5), and thus to be consistent, every metric should be prefixed by "mean", or none at all. Given the awkwardness of "mean precision", the most reasonable choice is to omit "mean" from average precision. We do not wish to take part in this argument, and use "MAP" and "MRR' simply because most researchers do. 47 It is fine to include additional metrics, but do not neglect the official metrics. 48 This likely started with BLEU scores in machine translation. of those rankings, for example, using any of those discussed in the previous section. And having quantified the effectiveness of results, it then becomes possible to make measurable progress in improving ranking models. We have our hill, we know how high up we are.
Although conceptually simple, the creation of resources to support reliable, large-scale evaluation of text retrieval methods is a costly endeavour involving many subtle nuances that are not readily apparent, and is typically beyond the resources of individual research groups. Fortunately, events such as the Text Retrieval Conferences (TRECs), organized by the U.S. National Institute for Standards and Technology (NIST), provide the organizational structure as well as the resources necessary to bring together multiple teams in community-wide evaluations. These exercises serve a number of purposes: First, they provide an opportunity for the research community to collectively set its agenda through the types of tasks that are proposed and evaluated; participation provides a barometer to gauge interest in emerging information access tasks. Second, they provide a neutral forum to evaluate systems in a fair and rigorous manner. Third, typical byproducts of evaluations include reusable test collections that are capable of evaluating systems that did not participate in the evaluation (more below). Some of these test collections are used for many years, some even decades, after the original evaluations that created them. Finally, the evaluations may serve as testbeds for advancing novel evaluation methodologies themselves; that is, the goal is not only to evaluate systems, but the processes for evaluating systems.
TREC, which has been running for nearly three decades, kicks off each spring with a call for participation. The evaluation today is divided into (roughly half a dozen) "tracks" that examine different information access problems. Proposals for tracks are submitted the previous year in the fall, where groups of volunteers (researchers from academia and industry) propose to organize a track. These proposals are then considered by a committee, and selected proposals define the evaluation tasks that are run. Over its history, TREC has explored a wide range of tasks beyond ad hoc retrieval, including search in a variety of different languages and over speech; in specialized domains such as biomedicine, and chemistry; different types of documents such as blogs and tweets; different modalities of querying such as filtering and real-time summarization; as well as interactive retrieval, conversational search, and other user-focused issues. For a general overview of different aspects of TREC (at least up until the middle of the first decade of the 2000s), the "TREC book" [Voorhees and Harman, 2005] edited by Voorhees and Harman provides a useful starting point.
Tracks at TREC often reflect emerging interests in the information retrieval community; explorations in TREC often set the agenda for the field and achieve significant impact beyond the academic ivory tower. Writing in 2008, Hal Varian, chief economist at Google, acknowledged that in the early days of the web, "researchers used industry-standard algorithms based on the TREC research to find documents on the web". 49 Another prominent success story of TREC is IBM's Watson question answering system that resoundingly beat two human champions in the quiz show Jeopardy! in 2011. There is a direct lineage from Watson, including both the techniques it used and the development team behind the scenes, to TREC question answering tracks held in the late 1990s and early 2000s.
Participation in TREC is completely voluntary with no external incentives (e.g., prize money), 50 and thus researchers "vote with their feet" is selecting tracks that are of interest to them. While track organizers begin with a high-level vision, the development of individual tracks is often a collaboration between the organizers and participants, aided by guidance from NIST. System submissions for the tasks are typically due in the summer, with evaluation results becoming available in the fall time frame. Each TREC cycle concludes with a workshop held on the grounds of the National Institute of Standards and Technology in Gaithersburg, Maryland, where participants convene to discuss the evaluation results and present their solutions to the challenges defined in the different tracks. 51 The cycle then begins anew with planning for the next year.
Beyond providing the overarching organizational framework for exploring different tracks at TREC, NIST also contributes evaluation resources and expertise, handling the bulk of the "mechanics" of the evaluation. Some of this was already discussed in Section 2.2: in most cases, 52 NIST assessors perform topic development, or the creation of the information needs, and provide the relevance 49 https://googleblog.blogspot.com/2008/03/why-data-matters.html 50 An exception is that sometimes a research sponsor (funding agency) uses TREC as an evaluation vehicle, in which case teams that receive receive funding are compelled to participate. 51 In the days before the COVID-19 pandemic, that is. 52 Unless specialized domain expertise is needed, for example, in biomedicine. assessments as well. Most of the NIST assessors are retired intelligence analysts, which means that assessing and otherwise drawing conclusions from information was, literally, their job. Topic development is performed in the spring time frame, based on initial exploration of the corpus used in the evaluation. To the extent possible, the assessor who created the topic (and wrote the topic statement) is the person who performs the relevance judgment (later that year, generally in the late summer to early fall time frame). This ensures that the judgments are as consistent as possible. To emphasize a point we have already made in Section 2.2: relevance judgments are the opinion of this particular person. 53
What do NIST assessors actually evaluate? In short, they evaluate the submissions (i.e., "runs") of teams who participated in the evaluation. For each topic, using a process known as pooling [Sparck Jones and van Rijsbergen, 1975 , Buckley et al., 2007 , runs from the participants are gathered, with duplicates removed, and presented to the assessor. To be clear, a pool is created for each topic. The most common (and fair) way to construct the pools is to select the top k results from each participating run, where k is determined by the amount of assessment resources available. This is referred to as top k pooling or pooling to depth k. Although NIST has also experimented with different approaches to constructing the pools [Voorhees, 2018] , top k pooling remains the most popular.
System results for each query are then presented to the assessor in an evaluation interface, who supplies the relevance judgments along the previously agreed scale (e.g., a three-way relevance grade). Note that in order to mitigate systematic biases, pooled results are not associated with the runs they are drawn from, so the assessor only sees (query, result) pairs and has no explicit knowledge of the source. After the assessment process completes, all judgments are then gathered to assemble the qrels for those topics, and these relevance judgments are used to evaluate the submitted runs (e.g., using one or a combination of the metrics discussed in the previous section).
Relevance judgments created from TREC evaluations are used primarily in one of two ways: First, they evaluate the effectiveness of systems that participated in a particular track. The evaluation of the submitted runs using the relevance judgments created from the pooling process accomplishes this goal, but the results need to be interpreted in a more nuanced way than just comparing the value of the metrics. Whether system differences can be characterized as significant or meaningful is more than just a matter of running a standard significance test, but must consider a multitude of other factors, including all the ones discussed in Section 2.2 and more [Sanderson and Zobel, 2005] . Details of how this is accomplished depend on the task and vary from track to track; for an interested reader, Voorhees and Harman [2005] is a good starting point. For more detail, in the TREC proceedings, each track comes with an overview paper written by the organizers that explains the task and the results in detail.
Second, and perhaps more impactful, the relevance judgments form a test collection that can be used as a standalone evaluation instrument by researchers beyond the confines of the original TREC evaluation that created them. These test collections can be used for years and even decades; for example, as we will describe in more detail in the next section, the test collection from the TREC 2004 Robust Track is still widely used today! In the context of using relevance judgments from a particular test collection, there is an important distinction between runs that participated in the evaluation vs. those that did not. There are two common use cases for using test collections: A team who participated in the TREC evaluation might use the relevance judgments to further investigate model variants or perhaps conduct ablation studies. A team who did not participate in the TREC evaluation might use the relevance judgments to evaluate a newly proposed technique, comparing it against runs submitted to the evaluation. In the former case, a variant technique is likely to retrieve similar documents as a submitted run, and therefore less likely to encounter unjudged documents-which, as we have previously mentioned, are treated as not relevant by standard evaluation tools (see Section 2.3). In the latter case, a newly proposed technique may encounter many unjudged documents, and thus score poorly-not necessarily because it was worse, but simplify because it was different. That is, the new technique surfaced documents that had not been previously retrieved (and thus never entered the pool to be assessed).
In other words, there is a danger that test collections encourage researchers to search only "under the lamplight", since the universe of judgments is defined by the participants of a particular evaluation (and thus represents a snapshot of the types of techniques that were popular at the time). Since many innovations work differently than techniques that came before, old evaluation instruments may not be capable of accurately quantifying effectiveness improvements of later techniques. As a simple example, if the pool were constructed exclusively from techniques based on exact term match, the resulting relevance judgments would be biased against systems that exploited semantic match techniques that did not depend on exact match signals. This is not idle speculation: provided evidence from the TREC 2019 Deep Learning Track that pools constructed from "traditional" keyword-based systems (i.e., based on exact match) led to biased results when evaluating systems based on neural networks, for precisely this reason. More specifically, the effectiveness of neural approaches were under-reported, making them seem less effective than they actually were. In general, old test collections may be biased negatively against new techniques, which is particularly undesirable because they may cause researchers to prematurely abandon promising innovations simply because the available evaluation instruments are not able to demonstrate their improvements.
Fortunately, IR researchers have long been cognizant of these dangers and take a variety of steps to guard against them. The most effective strategy is to ensure a rich and diverse pool, where runs adopt a variety of different techniques, and to actively encourage "manual" runs that involve a human in the loop (i.e., users interactively searching the collection to compile results). Since (obviously) humans do more than match keywords, manual runs increase the diversity of the pool. Furthermore, researchers have developed various techniques to assess the reusability of test collections, to fairly evaluate runs from systems that did not participate in the original evaluation [Zobel, 1998 , Buckley et al., 2007 . The literature describes a number of diagnostics, and test collections that pass this vetting are said to be reusable.
From a practical perspective, there are number of steps that researchers can take to sanity check their evaluation scores to determine if a run is actually worse, or simply different. One common technique is to compute and report the fraction of unjudged documents, as discussed in the previous section. If two runs have very different proportions of unjudged documents, this serves as a strong signal that one of those runs may not have been evaluated fairly. Another approach is to use a metric that directly quantifies the uncertainty in scores that can be attributed to unjudged documents, such as rank-based precision (RBP) [Moffat and Zobel, 2008] . 54 Obviously, different proportions of unjudged documents is a strong sign that effectiveness differences can be attributed to missing relevance judgments. However, an important note is that the absolute proportion of unjudged documents is not necessary a sign of unreliable evaluation results in itself. The critical issue is bias, in the sense of Buckley et al. [2007] : whether the relevance judgments represent a random (i.e., non-biased) sample of all relevant documents. Consider the case where two runs have roughly the same proportion of unjudged documents (say, half are unjudged). There are few firm conclusions that can be drawn in this situation without more context. Unjudged documents are inevitable, and even a relatively high proportion of unjudged isn't "bad" per se. This could happen, for example, when two runs that participated in an evaluation are assessed with a metric at a cutoff larger than the number of documents each run contributed to the pool. For example, the pool was constructed with top-100 pooling, but MAP is being measured to rank 1000. In such cases, there is no reason to believe that the unjudged documents are systematically biased against one run or the other. However, in other cases (for example, the bias introduced by systems based on exact term matching), there may be good reason to suspect the presence of systematic biases.
TREC, as a specific realization of the Cranfield paradigm, has been incredibly influential, both on IR research and more broadly in the commercial sphere; for example, see an assessment of the economic impact of TREC conducted in 2010 [Rowe et al., 2010] . TREC's longevity-now approaching its thirtieth year-is just one testament to its success. Another indicator of success is that the "TREC model" has been widely emulated around the world. Examples include CLEF in Europe and NTCIR and FIRE in Asia, which are organized in much the same way.
With this exposition, we have provided a high-level overview of modern evaluation methodology for text ranking under the Cranfield paradigm-covering inputs and outputs to the ranking model, how the results are evaluated, and how test collections are typically created. We conclude with a few words of caution already mentioned in the introductory remarks: The beauty of the Cranfield paradigm lies in a precise formulation of the ranking problem with a battery of quantitative metrics. This means that search can be tackled as an optimization problem using standard machine learning techniques. Beyond the usual concerns with overfitting, and whether test collections are realistic instances of information needs "in the wild", there is a fundamental question regarding the extent to which system improvements translates into user benefits. Let us not forget that the latter is the ultimate goal, because users seek information to "do something", e.g., decide what to buy, write a book report, find a job, etc. A well-known finding in information retrieval is that better search systems (as evaluated by the Cranfield methodology) might not lead to better user task performance as measured in terms of these ultimate goals; see, for example, Hersh et al. [2000] , Allan et al. [2005] . Thus, while evaluations using the Cranfield paradigm undoubtedly provide useful signal in characterizing the effectiveness of ranking models, they do not capture "the complete picture".
Supervised machine learning techniques require data, and the community is fortunate to have access to a number of valuable test collections, built over decades, for training text ranking models. In this section, we present a number of test collections that are commonly used to evaluate text ranking models. Our intention is not to exhaustively describe all the test collections used by every model in this survey, but to only focus on representative resources that have played an important role in the development of transformer-based ranking models.
When attempting to characterize and compare test collections, there are a number of key statistics to keep in mind:
• Size of the corpus, both in terms of the number of texts |C| and the average length of each document L(C).
• Size of the set of evaluation topics, both in terms of the number queries |q| and the average length of each query L(q).
• The amount of relevance judgments available, both in terms of positive and negative labels. We can quantify this in terms of the average number of judgments per query |J|/q as well as the number of relevant labels per query |Rel|/q.
A few key statistics of select test collections are summarized in Tables 2 and Table 3 . In more detail:
MS MARCO passage retrieval test collection. This dataset, originally released in 2016 [Nguyen et al., 2016] , deserves tremendous credit for jump-starting the BERT revolution for text ranking. Large, powerful neural network models need copious amounts of data in order to show off their capabilities, and thus BERT needed something like MS MARCO to illustrate its effectiveness for text ranking. We've already recounted the story in Section 1.2.5: Nogueira and Cho [2019] put these two important ingredients (model and data) together to make a "big splash" on the MS MARCO passage leaderboard. Today, this dataset is broadly used by researchers for a variety of information access tasks, and it has become a common starting point for building transformer-based ranking models. Many neural architectures are first fine-tuned with MS MARCO data before being fine-tuned further on task-specific data (see Section 3.5.5), even for domains that are as distant as biomedicine (see Section 3.7). Some experiments have even shown that ranking models fine-tuned on this test collection exhibit zero-shot relevance transfer capabilities, i.e., the models are effective in domains and on tasks Table 3 : Summary statistics for select queries and relevance judgments used by many text ranking models presented in this survey. For Robust04 we provide the average lengths of the title, narrative, and description fields of the topics.
without having been previously exposed to in-domain or task-specific labeled data (see Section 3.6.3 and Section 3.7).
While the original MS MARCO dataset was designed to support a variety of tasks, here we focus only on the passage retrieval task [Bajaj et al., 2018] . The corpus comprises 8.8 million passage-length extracts from web pages; these passages are typical of "answers" that many search engines today show at the top of their result pages. The information needs are anonymized natural language questions drawn from Bing's query logs, where users are specifically looking for an answer; queries with navigational and other intents were discarded. Since these questions are drawn from user queries "in the wild", they are often ambiguous, poorly formulated, and may even contain typographical and other errors. Nevertheless, these information needs reflect a more "natural" distribution of information needs that might be posed to, say, an intelligent assistant, compared to existing question answering datasets such as SQuAD [Rajpurkar et al., 2016] .
For each query, the test collection contains, on average, one relevant passage that has been annotated by a human editor. In the training set, there are a total of 532.7K (query, relevant passage) pairs over 502.9K unique queries. The development (validation) set contains 7437 pairs over 6980 unique queries. The test (evaluation) set contains 6837 queries, but relevance judgments are not publicly available; scores on the test queries can only be obtained via a submission to the official MS MARCO leaderboard. 55 The official evaluation metric is MRR@10.
One notable feature of this collection worth pointing out is the sparsity of judgments-there are many queries, but on average, only one relevant judgment per query. This stands in contrast with most test collections constructed by pooling, such as those from TREC evaluations. This has two important consequences:
1. Model training requires both positive as well as negative examples. For this, the task organizers have prepared "triples" files comprising (query, positive, negative) triples. However, these negative examples are pseudo-labels: they are drawn from candidates that have not been marked as non-relevant by human annotators. In other words, the negative examples have not been explicitly vetted by human annotators as definitely being not relevant. The absence of a positive label does not necessarily mean that the passage is not relevant. To our knowledge, however, there has not been a thorough exploration of the implications of this design.
2. As we will see in Section 3.2, the sparsity of the judgments holds important implications for the ability to properly assess the contribution of query expansion techniques. This is a known deficiency, but there may be other yet-unknown issues as well. The lack of "deep" judgments per query in part motivated the need for complementary evaluation data, which are supplied by the TREC 2019 Deep Learning Track (discussed below).
These flaws notwithstanding, it is difficult to exaggerate the important role that the MS MARCO dataset has played in advancing research in information retrieval and information access more 55 http://www.msmarco.org/ broadly. Never before had such a large and realistic dataset been made available to the academic research community; previously, such treasures were only available to researchers inside commercial search engine companies and other large organizations with substantial numbers of users engaged in information seeking. The organizers of the dataset and Microsoft deserve tremendous credit for their contributions to broadening the field.
MS MARCO document retrieval test collection. Although in reality the MS MARCO document test collection was developed in close association with the TREC 2019 Deep Learning Track (see below), and a separate MS MARCO document retrieval leaderboard was started only in August 2020, it makes more sense conceptually to structure the narrative in the order we present here.
The MS MARCO document retrieval test collection was created as a document retrieval counterpart to the passage retrieval task. The corpus, which comprises 3.2M web pages with URL, title, and body text, contains the source pages of the 8.8M passages from the passage corpus [Bajaj et al., 2018] . However, the alignment between the passages and the documents is imperfect, as the extraction was performed on web pages that were crawled at different times.
Relevance judgments were "transferred" from the passage judgments; that is, for a question, if a source web page contained a relevant passage, then the document was considered relevant. This data preparation created a systematic bias in that relevant information was artificially centered on a specific passage within the document, more so than they would occur naturally. For example, we are less likely to see a relevant document that contains short relevant segments scattered throughout the text; this has implications for evidence aggregation techniques that we discuss in Section 3.3.
In total, the MS MARCO document dataset contains 367K training queries and 5193 development queries; each query has exactly one relevance judgment. There are 5793 test queries, but relevance judgments are withheld from the public. As with the MS MARCO passage retrieval task, scores for the test queries can only be obtained by a submission to the leaderboard. The official evaluation metric is also MRR@10. Similar comments about the sparsity of relevance judgments, made in the context of the passage dataset above, apply here as well.
TREC 2019 Deep Learning Track. Due to the nature of TREC planning cycles, the organization of the Deep Learning Track at TREC 2019 predated the advent of BERT for text ranking. Coincidentally, though, it represented the first large-scale community evaluation that provided a comparison of pre-BERT and BERT-based ranking models, attracting much attention and participation from researchers.
From the methodological perspective, the track was organized to explore the impact of large amounts of training data, both on neural ranking models as well as learning-to-rank techniques, compared to "traditional" exact match techniques. Furthermore, the organizers wished to investigate the impact of different types of training labels, in particular, the sparse judgments (many queries but very few relevance judgments per query) typical of data gathered in an industry setting vs. dense judgments created by pooling (few queries but many more relevance judgments per query) that represent common practice in TREC and other academic evaluations. For example, what is the effectiveness of models trained on sparse judgments when evaluated with dense judgments?
The evaluation had both a document retrieval and a passage retrieval task; the organizers provided a list of results for reranking if participants did not wish to implement initial candidate generation themselves. The document corpus and the passage corpus used in the track were exactly the same the MS MARCO document corpus and MS MARCO passage corpus, respectively, discussed above. Despite the obvious connections, the document and passage retrieval task, were essentially considered separate (independent) evaluations.
Based on pooling, NIST assessors evaluated 43 queries based on a four-point scale. Statistics of the relevance judgements are provided in Table 3 . It is clear that these relevance judgments are not suitable for training neural ranking models (too few labeled examples), but they serve as a much richer test set. Since there are far more relevant documents per query, metrics such as MAP are meaningful. Since the relevance judgements are graded, metrics such as nDCG make sense. In contrast, given the sparse judgments in the original MS MARCO datasets, options for evaluation metrics are relatively limited. In particular, evaluation of a document retrieval task with MRR@10 is odd and rarely seen. [Voorhees, 2004] is widely considered one of the best "general purpose" ad hoc retrieval test collections available to academic researchers, with diverse relevance judgments that are able to fairly evaluate systems that did not participate in the original evaluation. It is large as academic test collections go, with high-quality relevance judgments provided within the same evaluation cycle. This contrasts with the typical evaluation practice where researchers concatenate test collections from multiple years, where there may be subtle year-to-year differences. For example, the composition of the judgment pools are qualitatively different, since they are constructed from participants from that particular year (using different techniques). Furthermore, the track drew many participants with diverse submissions, including manual runs that enriched the pool (see Section 2.4). Due to its age, the collection is particularly well-studied by researchers: conducted a meta-analysis that analyzed over 100 papers that used the collection, up until early 2019. 58 This resource provides the context for interpreting effectiveness results across entire families of approaches and over time. However, as it is publicly available, the downside is that the Robust04 test collection is particularly vulnerable to overfitting.
For training using relevance judgments from the collection, there is no standard agreed-upon split, but five-fold cross validation is the most common configuration. It is often omitted in papers, but researchers typically construct the splits by taking consecutive topic ids, e.g., the first fifty topics, the next fifty topics, etc.
Additional TREC newswire test collections. Beyond Robust04, there are two more recent newswire test collections that have been developed at TREC:
• Topics and relevance judgments from the TREC 2017 Common Core Track [Allan et al., 2017] , which used 1.8M articles from the New York Times Annotated Corpus. 59 Note that this evaluation experimented with a pooling methodology based on bandit techniques, which was found after-the-fact to have a number of flaws [Voorhees, 2018] , making it less reusable than desired. Evaluations conducted on this test collection should bear in mind this caveat.
• Topics and relevant judgments from the TREC 2018 Common Core Track [Allan et al., 2018] , which used a corpus of 600K articles from the TREC Washington Post Corpus. 60
Note that corpora for these two test collections are small by modern standards, and so they may not accurately reflect search scenarios today over large amounts of texts. In addition, both test collections are not as well-studied as Robust04. As a positive, this means there is less risk for overfitting, but this also means that there are fewer effective models to compare against.
TREC web test collections. There have been many evaluations at TREC that have focused on searching collections of web pages. In particular, the following three are commonly used:
• Unfortunately, there is no standard agreed-upon evaluation methodology (for example, training/test splits) for working with these test collections, and thus results reported in research papers are frequently not comparable. Additionally, unjudged documents are a concern, particularly with the ClueWeb collections, because the collection is large relative to the amount of assessment effort that was devoted to evaluating the pools. Furthermore, due to the barrier of entry in working with larger collections, there were fewer participating teams and thus less diversity in the run submissions.
We end this discussion with a caution, that as with any data for supervised machine learning, test collections can be abused and there is the ever-present danger of overfitting. When interpreting evaluation results, it is important to examine the evaluation methodology closely-particularly issues related to training/test splits and how effectiveness metrics are aggregated (e.g., if averaging is performed over topics from multiple years).
For these reasons, results from the actual evaluation (i.e., participation in that year's TREC) tend to be more "credible" in the eyes of many researchers than post hoc evaluations using the test collections, since there are more safeguards for introducing inadvertent biases. For example, TREC evaluations limit the total number of runs that are allowed from each group (typically three), which prevents a researcher from exploring many model variants, reporting only the best result, and neglecting to mention how many variants were examined (what is commonly known as "p-hacking"). Nevertheless, conclusions drawn from experiments on only one test collection should be avoided, and instead, ranking models should be evaluated on multiple test collections.
Although there are active explorations of alternatives (the entirety of Section 4 is devoted to this topic), most current applications of transformers for text ranking rely on keyword search in a multi-stage ranking architecture, which is the focus of Section 3. In this context, keyword search provides candidate generation, also called initial retrieval or first-stage retrieval. The results are then reranked by transformer-based models. Given the importance of keyword search in this context, we offer some general remarks to help the reader understand the role it plays in text ranking.
By keyword search or keyword querying, we mean a large class of techniques that rely on exact term matching to compute relevance scores between queries and texts from a corpus, nearly always with an inverted index (sometimes called inverted files or inverted lists); see Zobel and Moffat [2006] for an overview. This is frequently accomplished with bag-of-words queries, which refers to the fact that evidence (i.e., the relevance score) from each query term is considered independently. A bag-of-words scoring function can be cast into the form of Equation (1) in Section 1.2, or alternatively, as an inner product between two sparse vectors (where the vocabulary forms the dimension of the vector). However, keyword search does not necessarily imply bag-of-words queries, as there is a rich body of literature in information retrieval on so-called "structured queries" that attempt to capture relationships between query terms-for example, query terms that co-occur in a window or are contiguous (i.e., n-grams) Croft, 2004, 2005 ].
Nevertheless, one popular choice for keyword search today is bag-of-words queries with BM25 scoring (see Section 1.2), 64 but not all BM25 rankings are equivalent. In fact, there are many examples of putative BM25 rankings that differ quite a bit in effectiveness. One prominent example appears on the leaderboard of the MS MARCO passage retrieval task: a BM25 ranking produced by the Anserini system [Yang et al., 2017 scores 0.186 in terms of MRR@10, but the Microsoft BM25 baseline scores two points lower at 0.165.
Non-trivial differences in "BM25 rankings" have been observed by different researchers in multiple studies [Trotman et al., 2014 , Mühleisen et al., 2014 . There are a number of reasons why different implementations of BM25 yield different rankings and achieve different levels of effectiveness. First, BM25 should be characterized as a family of related scoring functions: Beyond the original formulation by Robertson et al. [1994] , many researchers have introduced variants, as studied by Trotman et al. [2014] , Mühleisen et al. [2014] , . Thus, when researchers refer to BM25, it is often not clear which variant they mean. Second, document preprocessing-which includes document cleaning techniques, stopwords lists, tokenizers, and stemmers-all have substantial impact on effectiveness. This is particularly the case with web search, where techniques for removing HTML tags, JavaScript, and boilerplate make a big difference [Roy et al., 2018] . The additional challenge is that document cleaning includes many details that are difficult to document in a traditional publication, making replicability difficult without access to source code. See for an effort to tackle this challenge via a common interchange format for index structures. Finally, BM25 (like most ranking functions) has free parameters that affect scoring behavior, and often researchers neglect to properly document these settings.
All of these issues contribute to differences in "BM25", but previous studies have generally found that the differences are not statistically significant. Nevertheless, in the context of text ranking with transformers, since the BM25 rankings are used as input for further reranking, prudent evaluation methodology dictates that researchers carefully control for these differences, for example with careful ablation studies.
In addition to bag-of-words keyword search, it is also widely accepted practice in research papers to present ranking results with query expansion using pseudo-relevance feedback as an additional baseline. As discussed in Section 1.2.2, query expansion represents one main strategy for tackling the vocabulary mismatch problem, to bring representations of queries and texts from the corpus into closer alignment. Specifically, pseudo-relevance feedback is a widely studied technique that has been shown to improve retrieval effectiveness on average; this is a robust finding supported by decades of empirical evidence. Query expansion using the RM3 pseudo-relevance feedback technique [Abdul-Jaleel et al., 2004] , on top of an initial ranked list of documents scored by BM25, is a popular choice (usually denoted as BM25 + RM3) .
To summarize, it is common practice to compare neural ranking models against both a bag-of-words baseline and a query expansion technique. Since most neural ranking models today (all of those discussed in Section 3 act as rerankers over a list of candidates, these two baselines also serve as the standard candidate generation approaches. In this way, we are able to isolate the impact and contributions of the neural ranking models.
A related issue worth discussing is the methodologically poor practice of comparisons to low baselines. In a typical research paper, researchers might claim innovations based on beating some baseline with a particular novel ranking model or approach. Such claims, however, need to be carefully verified by considering the quality of the baseline, in that it is quite easy to demonstrate improvement over low or poor quality baselines. This observation was made by Armstrong et al. [2009] , who conducted a survey and meta-analysis of research papers between 1998 and 2008 from major IR research venues that reported results on a diverse range of TREC test collections. Writing over a decade ago in 2009, they concluded: "There is, in short, no evidence that ad-hoc retrieval technology has improved during the past decade or more." The authors attributed much of the blame to the "selection of weak baselines that can create an illusion of incremental improvement" and "insufficient comparison with previous results". On the eve of the BERT revolution, conducted a similar meta-analysis and showed the pre-BERT neural ranking models were not any more effective than non-neural ranking techniques, at least with keyword queries on limited amounts of training data; but see a follow up in discussing BERT-based models. Nevertheless, the important takeaway message remains: when assessing the effectiveness of a proposed ranking model, it is necessary to also assess the quality of the comparison conditions, as it is always easy to beat a poor model.
There are, of course, numerous algorithmic and engineering details to building high-performance and scalable keyword search engines. However, for the most part, readers of this survey-researchers and practitioners interested in text ranking with transformers (and more generally, neural networks)-can treat keyword search as a "black box" using a number of open-source systems. From this perspective, keyword search is a mature technology that should be treated as infrastructure, or in modern "cloud terms", as a service. 65 It is safe to assume that this infrastructure can robustly deliver high query throughput at low query latency on arbitrarily large text collections: tens of milliseconds for modest collections, and at most a couple of hundred milliseconds for even web-scale collections; for example, see experimental results presented by Crane et al. [2017] . As we'll come to in Section 3.6, it is the inference latency of BERT and transformer models that forms the performance bottleneck in current architectures; candidate generation is very fast in comparison.
Today, only a few organizations-mostly commercial web search engines such as Google and Bing-deploy their own custom infrastructure for search. For most other organizations building and deploying search applications-in other words, practitioners of information retrieval-the open-source Lucene search library has emerged as the de facto standard solution, usually via either Elasticsearch or Solr, which are two search platforms that use Lucene at their core. Lucene powers search in production deployments at numerous companies, including Twitter, Bloomberg, Netflix, Comcast, Disney, Reddit, Wikipedia, and many more.
Academic information retrieval researchers have a long history of building and sharing search systems, dating back to Cornell's SMART system [Buckley, 1985] from the mid 1980s. Over the years, many open-source search engines have been built to aid in research, for example, to showcase new ranking models, query evaluation algorithms, or index organizations. An incomplete list, past and present, includes (in an arbitrary order) Lemur/Indri , Galago [Cartright et al., 2012] , Terrier [Ounis et al., 2006 , ATIRE [Trotman et al., 2012] , Ivory , JASS [Lin and Trotman, 2015] , JASSv2 [Trotman and Crane, 2019] , MG4J [Boldi and Vigna, 2005] , Wumpus, 66 and Zettair. 67 Until recently, however, few researchers used Lucene, which has created a chasm between information retrieval researchers and practitioners who build real-world search applications. Few in the latter category have even heard of any of the systems from the list above. And those in the former category paid little attention to search applications "in the real world".
As a result of this disconnect between research and practice, innovations from academic information retrieval research diffused into Lucene (and real-world search applications) at a painfully slow pace. For much of its existence, its default ranking model was an ad hoc variant of tf-idf that was demonstrably less effective than ranking models used in academic systems [Turtle et al., 2012] . BM25 was not added to Lucene until 2011, 68 nearly two decades after its introduction. However, in other respects, such as scalability and the ability to ingest heterogeneous content, Lucene is far ahead of any academic search engine.
Until recently, academic researchers perceived Lucene to be an inferior system ill-suited for cuttingedge research. This perception, however, begun to change a few years ago, in part because Lucene had "caught up". An evaluation exercise conducted in 2015 [Lin et al., 2016a] benchmarked several opensource search engines and demonstrated Lucene to be quite competitive in terms of both effectiveness and efficiency. Since then, there has been a resurgence of interest in adopting Lucene for information retrieval research, including a number of workshops that brought together like-minded researchers over the past few years [Azzopardi et al., 2017b,a] . The reasoning went: Why not combine the best of research and practice in order to bridge the divide between the two groups? Closer alignment means 65 Indeed, many of the major cloud vendors do provide search as a service. 66 http://www.wumpus-search.org 67 http://www.seg.rmit.edu.au/zettair/ 68 https://issues.apache.org/jira/browse/LUCENE-2959 that research innovation might have a short path to real-world impact; see, Grand et al. [2020] for an example case study.
One visible result of these efforts is Anserini [Yang et al., 2017 , an open-source toolkit built on Lucene designed to support replicable information retrieval research by providing many research-oriented features missing from Lucene, such as out-of-the-box support for a variety of common test collections (such as those discussed in the previous section). As the first author of this survey leads the development of Anserini and all of the co-authors use it in their own research, we are naturally partial to using the toolkit to provide candidate generation in a multi-stage ranking architecture. However, the reality is that Anserini has already been adopted to supply candidate generation for many of the ranking models we describe in this survey, and even papers that do not use Anserini directly adopt it as a basis for comparison. From the perspective of replicating results and providing easy-to-interpret comparisons, it makes sense to use Anserini as a starting point, but we would invite readers to try out the toolkit themselves to reach their own conclusions.
We conclude this section with some discussion of terminology used throughout this survey, where we have made efforts to be consistent in usage. As search is the most prominent instance of text ranking, our parlance is unsurprisingly influenced by information retrieval.
To start, IR researchers tend to favor the term "document collection" or simply "collection" over "corpus" (plural: corpora), which is more commonly used by NLP researchers. We use these terms interchangeably to refer to the "thing" containing the texts to be ranked.
In the academic literature, the meaning of the term "document" is overloaded: In one sense, it refers to the units of texts in the raw corpus. For example, news articles in a collection from the Washington Post, HTML files in a collection of web pages, a publication from a collection of scientific articles, etc.-these would all be considered documents. However, "documents" can also refer generically to the "atomic" unit of ranking (or equivalently, the unit of retrieval). For example, if Wikipedia articles are segmented into paragraphs for the purposes of ranking, each paragraph might be referred to as a document. This may appear slightly odd and may be a source of confusion as a researcher might continue to discuss document ranking, even though the documents are actually paragraphs.
In other cases, document ranking is explicitly distinguished from passage ranking-for example, there are techniques that retrieve documents from an inverted index (documents form the unit of retrieval), segment these documents into passages, score the passages, and then accumulate the scores to produce a document ranking, e.g., Callan [1994] . To add to the confusion, there are also examples where passages form the unit of retrieval, but passage scores are aggregated to rank documents, e.g., Hearst and Plaunt [1993] and Lin [2009] . We attempt to avoid this confusion by using the term "text ranking", leaving the form of the text underspecified and these nuances to be recovered from context. The compromise is that text ranking may sound foreign to the reader familiar with the IR literature. However, text ranking more accurately describe applications in NLP, e.g., ranking candidate in entity linking, as document ranking would sound especially odd in that context.
The information retrieval community often uses "retrieval" and "ranking" interchangeably, although the latter is much more precise. They are not, technically, the same: it would be odd refer to boolean retrieval as ranking, since such operations are manipulations of unordered sets. In a sense, retrieval is more generic, as it can be applied to situations where no ranking is involved, for example, fetching values from a key-value store. However, English lacks a verb that is more precise than to retrieve, in the sense of "to produce a ranking of texts" from, say, an inverted index, 69 and thus in cases where there is little chance for confusion, we continue to use the verbs "retrieve" and "rank" as synonyms.
Next, discussions about the positions of results in a ranked list can be a source of confusion, since rank monotonically increases but lower (numbered) ranks (hopefully) represent better results. Thus, a phrase like "high ranks" is ambiguous between rank numbers that are large (e.g., a document at rank 1000) or documents that are "highly ranked" (i.e., high scores = low rank numbers = good results).
The opposite ambiguity occurs with the phrase "low ranks". To avoid confusion, we refer to texts that are at the "top" of the ranked list (i.e., high scores = low rank numbers = good results) and texts that are near the "bottom" of the ranked list or "deep" in ranked list.
Finally, a note about the term "performance": Although the meaning of performance varies across different sub-disciplines of computer science, it is generally used to refer to measures related to speed such as latency, throughput, etc. However, NLP researchers tend to use performance to refer to output quality (e.g., prediction accuracy, perplexity, BLEU score, etc.). This can be especially confusing in a paper (for example, about model compression) that also discusses performance in the speed sense, because "better performance" is ambiguous between "faster" (e.g., lower inference latency) and "better" (e.g., higher prediction accuracy). In the information retrieval literature, "effectiveness" is used to refer to output quality, 70 while "efficiency" is used to refer to refer to properties such a latency, throughput, etc. 71 Thus, it is common to discuss effectiveness/efficiency tradeoffs. In this survey, our use of terminology is more closely aligned with the parlance in information retrieval-that is, we use effectiveness (as opposed to "performance") as a catch-all term for output quality and we use efficiency in the speed sense.
Okay, with the stage set and all these terminological nuances out of the way, we're ready to dive into transformers for text ranking! 70 Although even usage by IR researchers is inconsistent; there are still plenty of IR papers that use "performance" to refer to output quality. 71 Note that, however, efficiency means something very different in the systems community or the highperformance computing community.
The simplest and most straightforward formulation of text ranking is to convert the task into a text classification problem, and then sort the texts to be ranked based on the probability that each item belongs to the desired class. For information access problems, the desired class comprises those texts identified as relevant with respect to the user's information need (see Section 2.2), and so we can refer to this approach as relevance classification.
More precisely, the approach is to train a classifier that estimates the probability each text belongs to the "relevant" class, and then at ranking (i.e., inference) time sort the texts by these estimates. 72 This approach represents a direct realization of the Probability Ranking Principle, which states that documents should be ranked in decreasing order of the estimated probability of relevance with respect to the information need, first formulated in Robertson [1977] . Attempts to build models that directly perform ranking using supervised machine-learning techniques date back to the late 1980s [Fuhr, 1989] ; see also Gey [1994] . Both these papers describe formulations and adopt terminological conventions that would be familiar to readers today. To offer more historical context, today we would call relevance classification a pointwise learning to rank technique (see Section 1.2.3).
The first application of BERT to text ranking, by Nogueira and Cho [2019] , used BERT in exactly this manner. However, before describing this relevance classification approach in detail, we begin with a high-level overview of BERT. Our exposition is not meant to be tutorial in nature: rather, our aim is to highlight the aspects of the model that are important for explaining its applications to text ranking. had already shown BERT to be adept at classification tasks, and the adaptation by Nogueira and Cho-known as monoBERT-has proven to be a simple, robust, effective, and widely reproduced model for text ranking. It serves as the starting point for text ranking with transformers and provides a good baseline for subsequent ranking models that follow.
The progression of our presentation then takes the following course:
• The description of monoBERT introduces a key limitation of BERT for text ranking: the inability to handle long input sequences, and hence difficulty in ranking long texts (e.g., "fulllength" documents such as news articles). Researchers have proposed multiple solutions to overcoming this challenge, which are presented in Section 3.3. In particular, three of these ranking models-Birch [Akkalyoncu , BERT-MaxP [Dai and Callan, 2019b] , and CEDR [MacAvaney et al., 2019a]-are roughly contemporaneous and represent the "first wave" of neural ranking models that were capable of ranking longer documents, introduced a scant few months after the initial successes of Nogueira and Cho. • After presenting a number of BERT-based ranking models, we turn our attention to discuss the architectural context in which these models are deployed. A simple retrieve-then-rerank approach can be elaborated into a multi-stage reranking architecture with reranker pipelines, which Section 3.4 covers in detail. • The discussion of multi-stage ranking naturally leads to the question of "What's missing?" We answer this question by examining a number of techniques that can be applied outside the stages of multi-stage ranking architecture, but can nevertheless feed directly into its components. This provides the segue to document expansion, term importance prediction, and pretraining techniques in Section 3.5. • Next, we present a number of efforts that attempt to go beyond BERT in Section 3.6, to build ranking models that are faster (i.e., lower inference latency), that are better (i.e., higher ranking effectiveness), or that manifest an interesting tradeoff between effectiveness and efficiency. We cover ranking models that leverage relatively simple BERT variants, exploit knowledge distillation to train more compact student models, and other transformer architectures, including ground-up redesign efforts and reusing pretrained sequence-to-sequence models. These discussions set up a natural transition to ranking based on dense continuous representations, which is the focus of Section 4. • Finally, we wrap up this section with a discussion of domain-specific applications, covering the application of transformer architectures to the medical and biomedical domain as well as the legal domain (Section 3.7).
EA P8 E [CLS] T [CLS] [CLS] 
At its core, BERT (Bidirectional Encoder Representations from Transformers) is a neural network model for generating contextual embeddings for input sequences in English, with a multilingual variant (often called "mBERT") that processes input in over 100 different languages.
Here we focus only on the monolingual English model, but mBERT has been extensively studied as well Dredze, 2019, Pires et al., 2019] . BERT takes as input a sequence of tokens (more specifically, input vector representations derived from those tokens, more details below) and outputs a sequence of contextual embeddings, which provide context-dependent representations of the input tokens. 73 This stands in contrast to context-independent (i.e., static) representations, which include many of the widely adopted techniques that came before such as word2vec [Mikolov et al., 2013a] or GloVe [Pennington et al., 2014] .
The input-output behavior of BERT is illustrated in Figure 3 , where the input vector representations are denoted as:
and the output contextual embeddings are denoted as:
after passing through a number of transformer encoder layers.
In this way, BERT can be seen as a more sophisticated model with the same aims as ELMo [Peters et al., 2018] , from which BERT draws many important ideas: the goal of contextual embeddings is to capture complex characteristics of language (e.g., syntax and semantics) as well as how meanings vary across linguistic contexts (e.g., polysemy). The major difference is that BERT takes advantage of transformers [Vaswani et al., 2017] , as opposed to ELMo's use of LSTMs. BERT can be viewed as the "encoder half" of the full transformer architecture proposed by Vaswani et al. [2017] , which was designed for sequence-to-sequence tasks such as machine translation.
BERT is also distinguished from GPT [Radford et al., 2018] , another model from which it traces intellectual ancestry. If BERT can be viewed as an encoder-only transformer, GPT is the exact opposite: it represents a decoder-only transformer [Liu et al., 2018a] , or the "decoder half" of a full sequence-to-sequence transformer model. GPT is pretrained to predict the next word in a sequence based on its past history; in contrast, BERT uses a slightly different objective (although using similar intuitions based on language modeling), which leads to an important distinction discussed below. BERT and GPT are often grouped together as pretrained language models, although this characterization is somewhat misleading because, strictly speaking, a language model in NLP provides a probability distribution over arbitrary sequences of text tokens; see, for example [Chen and Goodman, 1996] . Transformers do far more than that! The significant advance that GPT and BERT represent over the original transformer formulation [Vaswani et al., 2017] is the use of self supervision in pretraining, whereas in contrast, Vaswani et al. started with random initialization of model weights and began directly training on labeled (input sequence, output sequence) pairs. This is an important distinction, as the insight of pretraining based on self supervision is arguably the biggest game changer in improving model output quality on a multitude of language analysis tasks. The beauty of self supervision is two-fold:
• Model optimization is no longer bound by the chains of labeled data. Self supervision means that the texts provide their own "labels" (in GPT, the "label" for a sequence of tokens is the next token that appears in the sequence), and that loss can be computed from the sequence itself (without needing any other external annotations). Since labeled data derive ultimately from human effort, removing the need for labels greatly expands the amount of data that can be fed to models for pretraining; computing power becomes the next bottleneck.
• Models optimized based on a self-supervised objective, without reference to any specific task, provide good starting points for further fine-tuning with task-specific labeled data. Empirical experiments have shown that a modest amount of labeled data is sufficient to achieve good effectiveness. Thus, the same pretrained model can provide the basis for performing multiple, different downstream tasks.
While they both exploit the insight of self supervision, GPT and BERT operationalize the idea in different ways. GPT uses a traditional language modeling objective, while BERT introduced the so-called "masked language model" (MLM) pretraining objective, which is inspired by the Cloze task [Taylor, 1953] , dating from over half a century ago. MLM is a fancy name for a fairly simple idea, not much different from peek-a-boo games that adults play with infants and toddlers: we randomly "cover up" (more formally, "mask") a token from the input sequence and ask the model to "guess" (i.e., predict) it, training with cross entropy loss. 74 The MLM objective explains the "B" in BERT, which stands for bidirectional: the model is able to use both a masked token's left and right contexts (preceding and succeeding contexts, respectively) to make predictions. In contrast, since GPT uses a language modeling objective, it is only able to use preceding tokens (i.e., the left context in a language written from left to right).
While the MLM objective was an invention of BERT, the idea of pretraining has a long history. ULMFiT (Universal Language Model Fine-tuning) [Howard and Ruder, 2018] likely deserves the credit for popularizing the idea of pretraining using language modeling objectives and then fine-tuning on task-specific data-the same procedure that has become universal today-but the application of pretraining in NLP can be attributed to Dai and Le [2015] . Tracing the intellectual origins of this idea even back further, the original inspiration comes from the computer vision community, dating back at least a decade [Erhan et al., 2009] .
Input sequences to BERT are usually tokenized with the WordPiece tokenizer [Wu et al., 2016] , although BPE [Sennrich et al., 2016 ] is a common alternative (used in GPT as well as RoBERTa [Liu et al., 2019c] ). These tokenizers have the aim of reducing the vocabulary space by splitting words into "subwords", usually in an unsupervised manner. For example, with the WordPiece vocabulary used by BERT, "biostatistics" becomes "bio" + "##statistics". The convention of prepending two hashes (##) to a subword indicates that it is "connected" to the previous word (i.e., in a language usually written with spaces, there is no space between the current subword and the previous one). In some cases, wordpieces do correspond to linguistically meaningful units, as in the example above. They can be helpful in separating words into their roots and inflections, as in splitting "walking" into "walk" and "##ing". This potentially helps the model learn a contextual embedding for the progressive tense, independent of the root verb. However, in other cases, wordpieces lack completely in any correspondence to linguistic reality, as in "adversarial" being tokenized into "ad", "##vers", "##aria", "##l". Nevertheless, the advantage of WordPiece tokenization (and related methods) is that a relatively 74 The actual procedure is a bit more complicated, but we refer the reader to the original paper for details. small vocabulary (e.g., 30,000 wordpieces) is sufficient to model large, naturally-occurring corpora that may have millions of unique tokens (if a simple method like tokenization by spaces is applied).
While BERT at its core converts a sequence of input embeddings into a sequence of contextual embeddings, in practice it is applied primarily to four types of tasks:
• Single-input classification tasks, for example, sentiment analysis.
• Two-input classification tasks, for example, detecting if two sentences are paraphrases of each other.
• Single-input labeling tasks, for example, named-entity recognition.
• Two-input labeling tasks, e.g., question answering, formulated as labeling the answer span in a text given a question.
The first token of every input sequence to BERT is a special token called [CLS]; the final representation of this special token is typically used for classification tasks. The [CLS] token is followed by the inputs: these are typically natural language sentences, but this need not be the case. For tasks involving a single input, another special delimiter token [SEP] is appended at the end of the input sequence. For tasks involving two inputs, both are packaged together into a single sequence, separated by the [SEP] token; the entire input sequence is also terminated with a [SEP] token. For labeling tasks over single inputs (e.g., named-entity recognition), the contextual embedding of the first subword is used to predict the correct tag that should be assigned to the token (e.g., in a standard BIO tagging scheme). Question answering (more generically, a labeling task involving two inputs) is treated in a conceptually similar manner, where the model attempts to tag the start and end of the answer span. These four tasks are illustrated in Figure 4 .
Pulling everything together, the input representation to BERT for each token comprises three components, shown at the bottom of Figure 3 :
• the token embedding from the WordPiece tokenizer (lookup from a dictionary); • the segment embedding, which is a learned embedding indicating whether the token belongs to first input (A) or the second input (B) in tasks involve two inputs (denoted E A and E B ) in Figure 3 ;
• the position embedding, which is a learned embedding capturing the position of the token in a sequence, allowing BERT to learn about relative positions between tokens.
The final input representation to BERT for each token comprises the element-wise summation of its token embedding, segment embedding, and position embedding. It is worth emphasizing that the three embedding components are summed, not assembled via vector concatenatation (this is a frequent point of confusion).
Representations of the input sequence in BERT are passed through a stack of transformer encoder layers to produce the output contextual embeddings. The number of layers, the hidden dimension size, and the number of attention heads are hyperparameters in the model architecture. However, there are number of "standard configurations", which we describe here. While the original work considered only the BERT Base and BERT Large variants, with 12 and 24 transformer encoder layers, respectively, later work [Turc et al., 2019] pretrained a wide variety of models with the help of knowledge distillation. The named BERT variants and their sizes are shown in Table 4 . While the smaller models are unlikely to be more effective than BERT Base or BERT Large , they contain substantially fewer parameters and are useful for exploring the effectiveness-efficiency tradeoffs (more in Section 3.6.1).
We conclude our high-level discussion of BERT by noting that its popularity is in no small part due to wise decisions by the authors (and approval by Google) to not only open source the model implementation, but also publicly release pretrained models (which were quite computationally expensive to pretrain from scratch). This led to rapid replication of the impressive results reported in the paper and provided other researchers with a reference implementation and basis for follow-up innovation. Today, the Transformers library 75 by Hugging Face has emerged as the de facto standard implementation of BERT as well as many transformer models, supporting both PyTorch [Paszke et al., 2019] and TensorFlow [Abadi et al., 2016] , the two most popular deep learning libraries today.
While open source (sharing code) and open science (sharing data and models) have become the norm in recent years, as noted by , the decision to share BERT wasn't necessarily a given. For example, Google could have elected not to share the source code or the pretrained models.
There are many examples of previous Google innovation that were shared in papers only, without a corresponding open source code release; MapReduce [Dean and Ghemawat, 2004] and the Google File System [Ghemawat et al., 2003] are two examples that immediately come to mind, although admittedly there are a number of complex considerations that factor into the binary decision to release code or not. In cases where descriptions of innovations in papers were not accompanied by source code, the community had to build its own open-source implementation from scratch (Hadoop in the case of MapReduce and the Google File System). This has generally impeded overall progress in the field because it required the community to rediscover many "tricks" and details from scratch that may not have been clear or included in the original paper. The community is fortunate that things turned out the way they did, and Google should be given credit for its openness. Ultimately, this led to an explosion of innovation in nearly all aspect of text analysis, including applications to text ranking. Figure 5 : Diagram of a retrieve-and-rerank architecture using transformers, which corresponds to the simplest instantiation of a multi-stage ranking architecture. In the candidate generation stage (also called initial retrieval or first-stage retrieval), candidate texts are retrieved from the document collection, typically with exact-match bag-of-words queries against inverted indexes. These candidates are then reranked with a transformer model such as BERT.
The task of a relevance classification is to estimate a score s i quantifying how relevant a candidate text d i is to a query q, which we denote as:
Before describing the details of how BERT is adapted for this task, let us first address the obvious question of where the candidate texts come from: Applying inference to every text in a corpus for every user query is (obviously) impractical from the computational perspective, not only due to costly neural network inference but also the linear growth of query latency with respect to collection size. While such a brute-force approach can be viable for small corpora, it quickly runs into scalability challenges. The standard solution to this challenges also answers the question of why Devlin et al.
[2019] never tackled text ranking in the original BERT paper, even though BERT was clearly designed for classification tasks (more below).
Although architectural alternatives are being actively explored by many researchers (the topic of Section 4), most applications of BERT for text ranking today adopt a retrieve-and-rerank approach, which is shown in Figure 5 . This represents the simplest instance of a multi-stage ranking architecture, which we detail in Section 3.4. In nearly all designs, candidate texts are generated using keyword search, usually with bag-of-words queries against inverted indexes (see Section 2.6). This retrieval stage is called candidate generation, initial retrieval, or first-stage retrieval, the output of which is a ranked list of texts, typically ordered by a scoring function based on exact term matches such as BM25 (see Section 1.2).
BERT inference is then applied to rerank these candidates to generate a score s i for each text d i in the candidates list. The BERT-derived scores may or may not be further combined or aggregated with other signals to arrive at the final scores. Nogueira and Cho [2019] used the BERT scores directly to rerank the candidates, thus treating the candidate texts as sets, but other approaches take advantage of, for example, the BM25 scores from the initial retrieval (more details later). Naturally, we expect that the ranking induced by these final scores have higher quality (for example, as measured by the metrics discussed in Section 2.3) than the scores from the initial retrieval stage.
Thus, most applications of BERT today to text ranking are actually performing reranking (but see Section 4). However, for expository clarity, we continue to refer to text ranking unless the distinction between ranking and reranking is important. The retrieval-and-rerank approach dates back to at least the 1960s [Simmons, 1965] and this architecture is mature and widely used (see Section 3.4).
This two-stage design also explains the major difference between Nogueira and Cho [2019] and the classification tasks described in the original BERT paper. only tackled text classification tasks that involved comparisons of two input texts (e.g., paraphrase detection), as opposed to text ranking, which requires multiple inferences using BERT. Recall that classification tasks involving two inputs are packed into a single sequence delimited by the [SEP] token, then fed to BERT for a single inference pass. Otherwise, relevance classification is a straightforward adaptation of BERT, which had already been demonstrated to work well other classification tasks.
E [CLS] T [CLS] [CLS] Figure 6 : The monoBERT ranking model adapts BERT for relevance classification by taking as input the query and a candidate text (surrounding by appropriate special tokens). The input vector representations comprise the element-wise summation of token embeddings, segment embeddings, and position embeddings. The output of the BERT model is a contextual embedding for each input token. The final representation of the [CLS] token is fed to a fully-connected layer that produces the relevance score s of that text with respect to the query.
Nogueira and Cho never gave their model a name, but for convenience we refer to it as monoBERT, which was coined later in Nogueira et al. [2019a] . The complete ranking model is shown in Figure 6 .
For the relevance classification task, monoBERT takes as input a sequence comprised of the following:
where q comprises the query tokens and d comprises tokens from the candidate text. This is the same input sequence configuration as in Figure 4 (b) for classification tasks involving two inputs. Note that the q tokens are taken verbatim from the user queries (or queries from the test collection); this detail will become important when we discuss the effects of feeding BERT different representations of the information need (e.g., "title" vs. "description" fields in TREC topics) in Section 3.3. The special tokens [CLS] and [SEP] are exactly those defined by BERT model, described in the previous section. In principle, the separator token [SEP] should be sufficient for the model to distinguish between tokens from different input sources (e.g., the query and candidate text in this case). In practice, however, found it helpful to explicitly add a type embedding to each input representation in the input sequence. For relevance classification, a type A embedding is added to query tokens and a type B embedding is added to document tokens. The final contextual representation of the [CLS] token is then used as input to a fully-connected layer that generates the document score s (more details below).
Since BERT was pretrained with sequences of tokens that have a maximum length of 512, tokens in an input sequence that is longer will not have a corresponding position embedding representation, and thus cannot be meaningfully fed to the model. Without position embeddings, BERT has no way to model the linear order and relative positions between tokens, and thus the model will essentially treat input tokens as a bag of words. In the datasets that Nogueira and Cho explored, this limitation was not an issue as the queries and candidate texts were shorter than this limit.
However, in general, the maximum sequence length of 512 tokens presents a challenge to using BERT for ranking longer texts. We set aside this issue for now and return to discuss solutions in Section 3.3, noting, however, that the simplest solution is to truncate the input. Since transformers exhibit quadratic complexity in both time and space with respect to input length [Kitaev et al., 2020] , it is common practice in production deployments to truncate the input sequence to a length that is shorter than the maximum allowed in order to manage latency. This might be a practical choice independent of BERT's length limitations.
The sequence of input tokens constructed from the query and a candidate text is then passed to BERT, which produces a contextual vector representation for each token (exactly as the model was designed to do). In monoBERT, the vector T CLS of the [CLS] token as input to a single-layer, fully-connected neural network to obtain a probability s i that the candidate d i is relevant to q; the contextual representations of the other tokens are not used. More formally:
where
is a bias term, and softmax(·) i denotes the i-th element of the softmax output. Since the last dimension of the matrix W is two, the softmax output has two dimensions (that is, the single-layer neural network has two output neurons), one for each class, i.e., "relevant" and "non-relevant".
BERT and the classification layer together comprise the monoBERT model. Following standard practices, the entire model is trained end-to-end for the relevance classification task using crossentropy loss:
where J pos is the set of indexes of the relevant candidates and J neg is the set of indexes of the non-relevant candidates obtained by the initial retrieval stage. Since the loss function takes into account only one candidate text at a time, this can be characterized as belonging to the family of pointwise learning to rank approaches [Liu, 2009 , Li, 2011 To be explicit, "training" here refers to the process of starting with a pretrained BERT model (i.e., a checkpoint provided by Google) and then fine-tuning it with task-specific supervised data. As this the standard way BERT is used in practice, we use "training BERT" in this sense for expository convenience (muddling the distinction between pretraining, training, and fine-tuning), since there is little risk of ambiguity. We refer the interested reader to the original paper by Nogueira and Cho for additional details, including hyperparameter settings.
There are two important details to note with respect to training monoBERT:
• The training loss makes no reference to the metric that is used to evaluate the final ranking (e.g., MAP), since each training example is considered in isolation; this is the case with all pointwise approaches. Thus, optimizing cross-entropy for classification may not necessarily improve an end-to-end metric such as mean average precision; in the context of ranking, this was first observed by Morgan et al. [2004] , who called this phenomenon "metric divergence".
• Texts that BERT sees at inference (reranking) time are different from examples fed to it during training. During training, examples are taken directly from a labeled examples, usually as part of an information retrieval test collection. In contrast, at inference time, monoBERT sees a set of candidates ranked by BM25, which may or not correspond to how the training examples were selected to begin with, and in some cases, we have no way of knowing since this detail may not have been disclosed by the test collection builders. Typically, during training, monoBERT is exposed to fewer candidates per query than at inference time, and thus the model is unable to learn an accurate distribution of BM25 scores across a pool of candidates varying in quality. Furthermore, the model usually does not see a realistic distribution of positive and negative examples. In some test collections, for example, positive and negative examples are balanced (i.e., equal numbers), so monoBERT is unable to accurately estimate the prevalence of relevant texts (i.e., build a prior) in BM25-scored texts (typically, far less than half of the texts from first-stage retrieval are relevant).
What's interesting is that, even without explicitly addressing these two details, the simple training scheme described above yields a relevance classifier that works well as a ranking model in practice. recall at cutoff 1000, which helps to quantify the upper bound effectiveness of the retrieve-then-rerank strategy. That is, if first-stage retrieval fails to return a relevant passage, the reranker cannot conjure one out of thin air. Since we do not have access to relevance judgments for the test set, it is only possible to compute recall for the development set.
The original monoBERT results, copied from Nogueira and Cho [2019] as row (2b) in Table 5 , is based on reranking baseline BM25 results provided by Microsoft, row (2a), with BERT Large . This is the result that in January 2019 kicked off the "BERT craze" for text ranking, as we've already discussed in Section 1.2. The effectiveness of IRNet in row (1), the best system right before the introduction of monoBERT, is also copied from Table 1 . The effectiveness of ranking with BERT Base is shown in row (2c), also copied from the original paper. We see that, as expected, a larger model yields higher effectiveness. Nogueira and Cho [2019] did not compute recall, and so the figures are not available for the conditions in rows (2a)-(2c).
Not all BM25 implementations are the same, as discussed in Section 2.6. The baseline BM25 results provided by Anserini (at k = 1000), row (3a), is nearly two points higher in terms of MRR@10 than the results provided by Microsoft's BM25 baseline, row (2a). Reranking Anserini results using monoBERT is shown in row (3b), taken from Nogueira et al. [2019a] ; reranking does not change recall. Improvements to first-stage retrieval do translate into more effective reranked results, but the magnitude of the improvement is not as large as the difference between Microsoft's BM25 and Anserini's BM25. The combination of Anserini BM25 + monoBERT Large , row (3b), provides a solid baseline for comparing BERT-based reranking models. These results can be replicated with PyGaggle, 76 which provides the current reference implementation of monoBERT.
Contrastive Experiments. We present two additional experiments to better understand the effectiveness of monoBERT under different conditions. These results were conducted with PyGaggle and have not been previously published.
Query expansion using pseudo-relevance feedback is a widely studied technique for improving retrieval effectiveness on average (see Section 2.6). Thus, it makes sense to apply pseudo-relevance feedback at the candidate generation stage prior to reranking by monoBERT. The effectiveness of keyword retrieval using BM25 + RM3 (in Anserini), a standard pseudo-relevance feedback baseline, is presented in row (4a) of Table 5 . Somewhat surprisingly, MRR@10 decreases with pseudo-relevance feedback. Further reranking with BERT, shown in row (4b), yields MRR@10 that is almost the same as reranking on BM25 results, row (3b). The recall between BM25 and BM25 + RM3 is also quite close. Thus, it appears that starting with a seemingly inferior candidate set (BM25 + RM3 vs. BM25), monoBERT is nevertheless able to identify relevant texts and bring them up into top-ranked positions.
What's going on here? These unexpected results can be attributed directly to artifacts of the relevance judgments in the MS MARCO passage retrieval test collection. It is well known that pseudo-relevance feedback has a recall enhancing effect, since the expanded query is able to capture additional terms that may appear in relevant texts. However, on average, there is only one relevant text per query; we have previously referred to these as sparse judgments (see Section 2.5 receive credit for improving recall. Thus, due to the sparsity of judgments, the MS MARCO passage retrieval test collection is fundamentally limited in its ability to detect effectiveness improvements from pseudo-relevance feedback.
We can better understand these effects by instead evaluating the same experimental conditions, but with the TREC 2019 Deep Learning Track passage retrieval test collection, which has far fewer topics, but many more judged passages per topic ("dense judgments", as described in Section 2.5). These results are shown in Table 6 , where the rows have been numbered in the same manner as Table 5 . We can see that these results support our explanation above: in the absence of BERT-based reranking, pseudo-relevance feedback does indeed increase effectiveness, as shown by row (3a) vs. row (4a).
In particular, recall increases by around five points. Furthermore, the increase in the quality of the candidates does improve end-to-end effectiveness after reranking, row (3b) vs. row (4b), although the magnitude of gain is smaller than the impact of pseudo-relevance feedback over simple bag-of-words. An important takeaway here is the importance of recognizing the limitations of a particular evaluation instrument (i.e., the test collection) and when an experiment exceeds its assessment capabilities.
Having explored the impact of different candidate generation approaches, we next tackle two obvious questions (both on the development set of the MS MARCO passage retrieval test collection) that are raised by the retrieve-then-rerank architecture:
1. How does retrieval depth k affect end-to-end effectiveness? 2. Do exact match scores from initial retrieval contribute to end-to-end effectiveness?
The first question is answered in Figure 7 , where we show end-to-end effectiveness (MRR@10) of monoBERT with BM25 supplying different numbers of candidates to rerank. It is no surprise that end-to-end effectiveness increases as retrieval depth k increases, although there is clearly diminishing returns: going from 1000 hits to 10000 hits increases MRR@10 from 0.372 to 0.377. Further increasing k to 50000 does not measurably change MRR@10 at all (same value). The effectiveness curve does not appear to be convex, which is an interesting empirical finding: To phrase differently, it is not the case (at least out to 50000 hits) that effectiveness decreases with more candidates beyond a certain point. This behavior might be plausible because we are feeding BERT increasing worse results, at least from the perspective of BM25 scores. However, it appears that BERT is not "confused" by such texts. Since latency increases linearly with the number of the candidates processed (in the absence of intra-query parallelism), this finding has important implications for real-world deployments: system designers should simply select the largest k practical given their available hardware budget and latency targets. There does not appear to be any danger in considering k values that are "too large" (which would be the case if the effectiveness curve were convex, thus necessitating more nuanced tuning to operate at the optimal setting). In other words, the tradeoff between effectiveness and latency is straightforward to manage.
With respect to the second question, one obvious approach to combining evidence from initial BM25 retrieval scores and BERT is linear interpolation, whose usage in document ranking dates back to at least the 1990s [Bartell et al., 1994] :
where s i is the final document score, s BM25 is the BM25 score, s BERT is the BERT score, and α ∈ [0..1] is a weight the indicates their relative importance. Since BERT scores are ∈ [0, 1], we also normalize BM25 scores to be in the same range via linear scaling: Figure 9 : Side-by-side comparisons between high-level architectures of the two main classes of pre-BERT neural ranking models with monoBERT. In monoBERT, all-to-all attention at each layer in the transformer captures interactions between terms from the query and terms from the candidate text, as well as interactions within terms from the query and terms from the candidate text.
where s is the original score,ŝ is the normalized score, and s max and s min are the maximum and minimum scores, respectively, in the ranked list. The results are shown in Figure 8 , which shows that MRR@10 decreases as we increase the weight placed on BM25 scores. This result suggests that exact term matching scores, at least using this simple approach, do not provide any relevance signals that is not already captured by BERT.
Model Extensions. On top of the basic monoBERT design, researchers have explored a few extensions, primarily aiming to improve the training procedure. For example, Zhang et al. [2020d] proposed a method for training monoBERT with weak supervision by using reinforcement learning to select (anchor text, candidate text) pairs during training. In this approach, relevance judgments are used to compute the reward guiding the selection process, but the model does not use the judgments directly. To apply the trained model to a target collection, the authors trained a learning to rank method using coordinate ascent with features consisting of the first-stage retrieval score and monoBERT's [CLS] vector. Zhang et al. found that these extensions improved over prior weak supervision approaches used with neural rankers [Dehghani et al., 2017 , MacAvaney et al., 2019b .
Also hoping to improve training, MacAvaney et al.
[2020d] investigated whether monoBERT can benefit from a training curriculum ] in which the model is presented with progressively more difficult training examples as training progresses. Using BM25 to estimate a training example's difficulty, they found that curriculum learning can significantly improve monoBERT's effectiveness.
Another innovation building on monoBERT is the work of Zheng et al. [2020] , who extended the pre-BERT NPRF (Neural Pseudo Relevance Feedback) approach to take advantage of BERT-based relevance classification. In this approach, the first-stage retrieval results are first reranked with monoBERT to identify feedback documents from among the top results (i.e, documents likely to be relevant). Next, monoBERT is used to score snippets from the feedback documents in order to identify feedback snippets. Given these snippets, each document's relevance score is computed as a combination of the document's similarity to the original query and to each feedback snippet. Zheng et al. found that this approach significantly improved effectiveness over the standalone BERT model, but with decreased efficiency due to the new snippet-document comparisons introduced.
Discussion and Analysis. Reflecting on the results presented above, it is quite remarkable how monoBERT offers a simple yet effective solution the text ranking problem (at least for texts that fit within its sequence length restrictions). The simplicity of the model has contributed greatly to its widespread adoption. These results have been widely reproduced and can be considered robust findings-for example, different authors have achieved comparable results across different implementations and hyperparameter settings (some variant of monoBERT serves as the baseline for many of the papers cited throughout this survey). Indeed, the monoBERT model has emerged as the baseline for transformer-based approaches to text ranking.
What is the relationship between BERT and previous "pre-BERT" neural ranking models, such as those discussed in Section 1.2.4? Figure 9 tries to highlight important architectural differences: for convenience, we repeat the high-level designs of the pre-BERT representation-based and interactionbased neural ranking models from Figure 1 (Section 1.2.4). In general, interaction-based approaches (middle) are more effective than representation-based approaches (left) because the similarity matrix explicitly models matches (both exact and "soft" semantic) between individual terms as well as sequences of terms (for some models) from the query and the candidate text. In BERT, all-to-all query-text term interactions (as well as interactions between terms in the query and terms in the text) are captured by multi-headed attention at each layer in the transformer. Attention serves as one-size-fits-all approach to extracting signal from term interactions, replacing the various techniques used by pre-BERT interaction-based models, e.g., different pooling techniques, convolutional filters, etc.
It appears that BERT does not require any specialized neural architectural components to model different aspects of relevance between a query and a text, since each layer of the transformer is homogeneous and the same exact architecture is used for a range of other language analysis task. 77
There has been no shortage of research that attempts to reveal insights about how BERT "works". Typically, this is accomplished through visualization techniques (for example, of attention and activation patterns) and probing classifiers. For example, Clark et al.
[2019] categorized a few frequently observed patterns such as attending to delimiter tokens and specific position offsets, and was able to identify attention heads that correspond to linguistic notions (e.g., verbs attending to direct objects). Kovaleva et al. [2019] specifically focused on self-attention patterns and found that a limited set of attention patterns are repeated across different heads, indicating that the model is overparametrized. Indeed, manually disabling attention in certain heads leads to effectiveness improvements in some NLP tasks [Voita et al., 2019] . Nevertheless, they also found evidence of attention patterns that corresponds to semantic relations. The analyses of Tenney et al. [2019] led to the broader claim that "BERT rediscovers the classical NLP pipeline", in that the model represents part-of-speech tagging, parsing, named-entity recognition, semantic role labeling, and coreference (in that order) in an interpretable and localizable way. While these studies begin to shed light on the inner workings of BERT, they do not specifically examine information access tasks, so they offer limited insight on how notions of relevance are captured by BERT.
Information retrieval researchers, however, have attempted to specifically examine relevance matching by BERT in ranking tasks [Padigela et al., 2019 , Qiao et al., 2019 , Zhan et al., 2020a . While progress has been made, it is fair to say that we still do not completely understand how BERT works for text ranking. Qiao et al. [2019] argued that BERT should be understood as an "interaction-based sequence-to-sequence matching model" that prefers semantic matches between paraphrase tokens. Furthermore, the authors also found that BERT's relevance matching behavior differs from neural rankers that are trained from user clicks in query logs. The analysis of Zhan et al. [2020a] , however, arrived at a slightly different conclusion, arguing that the lower layers of BERT focus primarily on extracting representations, while the higher layers capture interaction signals to ultimately predict relevance. Thus, BERT shows evidence of combining elements from both representation-based as well as interaction-based models.
One notable limitation of monoBERT is that it does not offer an obvious solution to the input length limitations of BERT. Nogueira and Cho [2019] did not have this problem because the test collections they examined did not contain texts that overflowed this limit. Thus, monoBERT is limited to ranking paragraph-length passages, not longer documents (e.g., news articles) as is typically found in most ad hoc retrieval test collections. However, the design of BERT's architecture and the pretraining configuration means that it has difficulty handling input sequences longer than 512 tokens, both from the perspective of model effectiveness and computational requirements on present-day hardware. Let us begin by understanding in more detail what the issues are.
Since BERT was pretrained with only input sequences up to 512 tokens, learned position embeddings for token positions past 512 are not available. Because position embeddings inform the model about the linear order of tokens and their relative positions, if the input sequence lacks this signal, then 77 There is, however, recent work suggesting that BERT might benefit from explicit exact match cues conveyed using marker tokens [Boualili et al., 2020] . However, the authors reported results only with BERT base (with MRR@10 scores much lower than ones reported here), and thus it is unclear if such explicit cues are effective with a more competitive baseline.
everything the model has learned about the structure of language is lost (i.e., the input will essentially be treated as a bag of words). Henderson [2020] explained this by pointing out that BERT can be thought of as a "bag of vectors", where structural cues come only from the position embeddings. This means that the vectors in the bag are exchangeable, in that renumbering the indices used to refer to the different input representations will not change the interpretation of the representation (provided that the model is adjusted accordingly as well). While it may be possible to learn additional position embeddings during fine-tuning with sufficient training data, this does not seem like a practical general-purpose solution.
From the computational perspective, the all-to-all nature of BERT's attention patterns at each transformer encoder layer means that it exhibits quadratic complexity in both time and space with respect to input length [Kitaev et al., 2020] , and so simply throwing more hardware at the problem (e.g., GPUs with more RAM) is not a practical solution. Instead, Kitaev et al. [2020] proposed the "Reformer", which replaces standard dot-product attention by a design based on locality-sensitive hashing to efficiently compute the attention only against the most similar tokens, thus reducing model
where L is the length of the sequence. Similarly, Beltagy et al.
[2020] provided experimental results characterizing resource consumption on present-day hardware with increasing sequence lengths. Their solution, dubbed "Longformer", addressed the blow-up in resource usage by sparsifying the all-to-all attention patterns in the basic transformer design. While there may indeed be merit to both approaches, these designs have not yet to date been tested on document ranking problems.
The length limitation of BERT (and transformers in general) breaks down into two distinct but related issues for text ranking:
Training. For training, it is unclear what to feed to the model. The key issue is that relevance judgments for document ranking (e.g., from TREC collections) are provided at the document level, i.e., they are annotations on the document as a whole. Obviously, a judgment of "relevant" comes from a document containing "relevant material", but it is unknown how that material is distributed throughout the document. For example, there could be a relevant passage in the middle of the document, a few relevant passages scattered around the document, or the document may be relevant "holistically" when considered in its entirety, but without any specifically relevant passages. If we wish to explicitly model different relevance grades (e.g., relevant vs. highly relevant), then this "credit assignment" problem becomes even more challenging.
During training, if the input sequence (i.e., document plus the query and the special tokens) exceeds BERT's length limitations, it must be truncated somehow, lest we run into exactly the issues discussed above. Since queries are usually shorter than documents, and it make little sense to truncate the query, we must sacrifice terms from the document text. While we could apply heuristics, for example, to feed BERT only spans in the document that contain query terms or even disregard this issue completely (see Section 3.3.2), there is no guarantee that training passages from the document fed to BERT are actually relevant. Thus, training will be noisy at best.
Inference. At inference time, if a document is too long to feed into BERT in its entirety, we must decide how to preprocess the document. We could segment it into chunks, but there are many design choices: For example, fixed-width spans or natural units such as sentences? How wide should these segments be? Should they be overlapping? Furthermore, applying inference over different chunks from a document still requires some method for aggregating evidence.
It is possible to address the inference challenge by aggregating either passage scores or passage representations. Methods that use score aggregation predict a relevance score for each chunk, which are then aggregated to produce a document relevance score (e.g., by taking the maximum score). Methods that perform representation aggregation first combine passage representations before predicting a relevance score. If these aggregation techniques are properly designed, even though each passage is independently processed, the complete ranking model that uses BERT as a component can be differentiable and thus amenable to end-to-end training via back propagation. This solves the training challenge as well, primarily by letting the model figure out how to allocate "credit" by itself.
Breaking this "length barrier" in transitioning from passage ranking to full document ranking was the next major advance in applying BERT to text ranking. This occurred with three proposed models that were roughly contemporaneous, dating to Spring of 2019, merely a few months after monoBERT:
Birch [Akkalyoncu , which was first described in Yang et al. [2019e] , BERT-MaxP [Dai and Callan, 2019b] , and CEDR [MacAvaney et al., 2019a] . Interestingly, these three models took different approaches to tackling the challenges described above. Below, we present each of them in detail, and also present two more recent developments: PCGM [Wu et al., 2020b] and PARADE . All of these ranking models are still based on BERT at their cores; we discuss efforts to move beyond BERT in Section 3.6.
The solution provided by Birch [Akkalyoncu can be summarized as follows:
• Avoid the training problem entirely by exploiting data where length issues don't exist, and then transferring those relevance matching models into the domain/task of interest. • For the inference problem, convert the task of estimating document relevance into the task of estimating the relevance of individual sentences and then aggregating the resulting scores.
In short, Birch solved the training problem above by simply avoiding it. An earlier work [Yang et al., 2019e ] that eventually gave rise to Birch first examined the task of ranking tweets, using test collections from the TREC Microblog Tracks [Ounis et al., 2011 , Soboroff et al., 2012 , Lin and Efron, 2013 . These evaluations focused on information seeking in a microblogging context, where users desire relevant tweets with respect to an information need at a particular point in time.
As tweets are short (initially 140 characters, now 280 characters), they completely avoid the length issues we discussed above.
Not surprisingly, fine-tuning monoBERT on tweet data led to large and statistically significant gains on ranking tweets. However, Yang et al. [2019e] discovered that a monoBERT model fine-tuned with tweet data was also effective for ranking documents from a newswire corpus. This was a surprising finding: despite similarities in the task (both are ad hoc retrieval problems), the domains are completely different. Newswire articles comprise well-formed and high-quality prose written by professional journalists, whereas tweets are composed by social media users, often containing mispellings, ungrammatical phrases, and incoherent meanings, not to mention genre-specific idiosyncrasies such as hashtags and @-mentions.
In other words, Yang et al. [2019e] discovered that, for text ranking, BERT seems to have very strong domain transfer effects for relevance matching. Training on tweet data and performing inference on articles from a newswire corpus is an instance of zero-shot cross-domain learning, since the model had never been exposed to annotated data from the specific task. 78
This domain-transfer discovery was later refined by Akkalyoncu in Birch. To compute document relevance s f , inference is applied to each individual sentence in the document, and then the top n scores are combined with the original document score s d (i.e., for first-stage retrieval) as follows:
where s i is the score of the i-th top scoring sentence according to BERT. Inference on individual sentences is exactly the same as in monoBERT, where the input to BERT is comprised of the concatenation of the query q and a sentence p i ∈ D into the sequence:
In other words, the relevance score of a document comes from the combination of the original candidate document score s d (e.g., from BM25) and evidence contributions from the top sentences in the document as determined by the BERT model. The parameters α and the w i 's can be tuned via cross-validation.
Results and Analysis. Birch results are reported in Table 7 with BERT Large on the Robust04, Core17, and Core18 test collections (see Section 2.5), with metrics directly copied from Akkalyoncu To be explicit, the query tokens q fed into BERT come from the "title" portion of the TREC topics (see Section 2.2), i.e., short keyword phrases. This distinction will become important when we discuss Dai and Callan [2019b] next. The results in the table are based on reranking the top k = 1000 candidates using BM25 from Anserini for first-stage retrieval, using the topic titles as bag-of-words queries. See the authors' paper for detailed experimental settings. Note that none of these collections were used to fine-tune the BERT relevance models; the only learned parameters are the weights in Eq. (17).
The top row shows the BM25 + RM3 query expansion baseline. The remaining blocks display the ranking effectiveness of our models on Robust04, Core17, and Core18. Each row describes an experimental condition: nS indicates that inference was performed using the top n scoring sentences from each document. Up to three sentences were considered; the authors reported that more sentences did not yield any improvements in effectiveness. The notation in parentheses describes the fine-tuning procedure: MB indicates that BERT was fine-tuned on data from the TREC Microblog Track; MS MARCO indicates that BERT was fine-tuned on data from the MS MARCO passage retrieval test collection; MSMARCO → MB refers to a model that was first fine-tuned on MS MARCO and then further fine-tuned on MB. Table 7 also includes results of significance testing using paired t-tests, comparing each condition with the BM25 + RM3 baseline. Statistically significant differences with are denoted by the symbol † , at the p < 0.01 level, with appropriate Bonferroni corrections for multiple hypothesis testing.
Birch, fine-tuned on microblog data (MB) alone significantly outperforms the BM25 + RM3 baseline for all three metrics on Robust04. On Core17 and Core18, significant increases in MAP are observed as well (and other metrics in some cases). In other words, the relevance classification model learned from tweets successfully transfers over to news articles despite the large aforementioned differences in domain.
Interestingly, Akkalyoncu reported that fine-tuning on MS MARCO alone yields smaller gains over the baselines compared to fine-tuning on tweets. The gains in MAP are statistically significant for Robust04 and Core17, but not Core18. In her unpublished thesis, Akkalyoncu conducted experiments that provided an explanation: this behavior is attributable to mismatches in input text length between the training and test data. The average length of the tweet training examples is closer to the average length of sentences in Robust04 than the passages in the MS MARCO passage corpus (which are much longer). Simply truncating MS MARCO passages to the average length of sentences in Robust04 yielded a large boost in effectiveness: 0.3300 MAP on Robust04. While this result is still below fine-tuning only with tweets, simply truncating MS MARCO passages also degrades the quality of the dataset, in that the operation could discard the relevant portions of the passage, thus leaving behind an incorrect relevance label.
The best condition in Birch is to fine-tune with MS MARCO passages, and then fine-tune (again) with tweet data, which yields effectiveness that is higher than fine-tuning with each dataset alone. Looking across all fine-tuning configurations of Birch, it appears that the top-scoring sentence of each candidate document alone is a good indicator of document relevance. Additionally considering the second ranking sentence yields at most a minor gain, and in some cases, adding a third actually causes effectiveness to drop. In all cases, however, contributions from BM25 scores remain important-the model places non-negligible weight on α in Eq. (17). This result does not appear to be consistent with the monoBERT experiments described in Figure 8 , which shows that beyond defining the top k candidates fed to monoBERT, the BM25 scores do not provide any additional relevance signal, and in fact interpolating BM25 scores hurts effectiveness. The two models, of course, are evaluated on different test collections, but the question of whether exact term matching scores are still necessary for relevance classification with BERT remains open. We have shown that monoBERT effectiveness remains unchanged all the way out to reranking k = 50000 candidates, where no doubt monoBERT is fed passages that have poor BM25 scores with respect to the query. Clearly, monoBERT is not "confused" by these low quality texts, as to need evidence from explicit BM25 scores. Why, then, does BERT need similar help in Birch?
The thesis of Akkalyoncu Yilmaz [2019] described additional ablation experiments that revealed interesting insights about the behavior of BERT for document ranking. It has long been known (see discussion in Section 1.2.2) that modeling the relevance between queries and documents requires a combination of exact term matching (i.e., matching the appearance of query terms in the text) as well as "semantic matching", which encompasses attempts to capture a variety of linguistic phenomena including synonym, paraphrases, etc. What is the exact role that each plays in BERT? To answer this question, Akkalyoncu performed an ablation experiment where all sentences that contain a query term are discarded; this has the effect of eliminating all exact match signals and forces BERT to rely only on semantic match signals. As expected, effectiveness was much lower, reaching only 0.3101 MAP on Robust04 in the best model configuration, but the improvements over the BM25 + RM3 baseline (0.2903 MAP) remain statistically significant. This result suggests that with BERT, semantic matching signals contribute an important component to overall the model for relevance matching.
As an anecdotal example, for the query "international art crime", in one relevant document, the following sentence was identified as the most relevant: "Three armed robbers take 21 Renaissance paintings worth more than $5 million from a gallery in Zurich, Switzerland." Clearly, this sentence contains no terms from the query, but yet provides information relevant to the information need. An analysis of the attention patterns shows strong association between "art" and "paintings" and between "crime" and "robbers" in the different transformer encoder layers. Here, we can see that BERT accurately captures semantically important matches for the purposes of modeling query-document relevance, providing complementary evidence supporting the conclusion above.
To provide some broader context for the level of effectiveness achieved by Birch: Akkalyoncu claimed to have reported the highest known MAP at the time of publication on the Robust04 dataset. This assertion appears to be supported by the meta-analysis of , who analyzed over 100 papers up until early 2019 and placed the best neural model at 0.3124 . These results also exceeded the previous highest known score of 0.3686, a non-neural method based on ensembles [Cormack et al., 2009] reported in 2009. On the same dataset, CEDR [MacAvaney et al., 2019a] (which we discuss in Section 3.3.3) achieved a slightly higher nDCG@20 of 0.5381, but did not report MAP. BERT-MaxP (which we discuss in Section 3.3.2) reported 0.529 nDCG@20. It seems clear that the "first wave" of text ranking models based on BERT was able to outperform pre-BERT models and at least match the best non-neural techniques known at the time. 79 These scores, in turn, have been bested by even newer ranking models such as PARADE (Section 3.3.4) and monoT5 (Section 3.6.3). The best Birch model also achieved a higher MAP than the best TREC submission that does not make use of past labels or human intervention for both Core17 and Core18, although it is true that both test collections are relatively new and thus have yet to receive much attention from researchers.
Takeaway Lessons. Summarizing, there are two important takeaways to Birch:
1. BERT exhibits strong zero-shot cross-domain relevance classification capabilities. That is, we can train a BERT model using relevance judgments from one domain (e.g., tweets) and directly apply the model to relevance classification in a different domain (e.g., newswire articles) and achieve a high-level of effectiveness.
2. The relevance score of the highest-scoring sentence in a document is a good proxy for the relevance of the entire document. In other words, it appears that document-level relevance can be accurately estimated by considering only a few top sentences.
The first point illustrates the power of BERT, attributable to wonders of pretraining. The finding with Birch is consistent with other demonstrations of BERT's zero-shot capabilities, for example, in question answering [Petroni et al., 2019] . We return to elaborate on this observation in Section 3.5.5 in the context of additional pretraining and "stage-wise" fine-tuning techniques, in Section 3.6.3 in the context of ranking with sequence-to-sequence models, and again in Section 3.7 in the context of domain-specific applications.
The second point is consistent with previous findings in the information retrieval literature as well as the BERT-MaxP model that we describe next. We defer a more detailed discussion of this takeaway after presenting that model.
Another solution to the length limitations of BERT is offered by Dai and Callan [2019b] , which can be summarized as follows:
• For training, don't worry about it! Segment documents into overlapping passages: treat all segments from a relevant document as relevant and all segments from a non-relevant document as not relevant.
• For the inference problem, segment documents in the same way, estimate the relevance of each passage, and then perform simple aggregation of the passage relevance scores (see below) to arrive at the document relevance score.
In more detail, documents are segmented into passages using a 150-word sliding window with a stride of 75 words. Window width and stride length are hyperparameters, but Dai and Callan [2019b] did not report experimental results exploring the effects of different settings. Inference on the passages is the same as in Birch and in monoBERT, where for each passage p i ∈ D, the following sequence is constructed and fed to BERT:
where q is the original question. The [CLS] token is then fed into a fully-connected layer (exactly as in monoBERT) to produce a score s i for passage p i . 80 The passage relevance scores {s i } are then aggregated to produce the document relevance score s d according to one of three approaches:
• BERT-MaxP: take the maximum passage score as the document score, i.e., s d = max s i • BERT-FirstP: take the score of the first passage as the document score, i.e., s d = s 1 .
• BERT-SumP: take the sum of all passage scores as the document score, i.e., s d = i s i .
Another interesting aspect of this work is an exploration of different query representations that are fed into BERT. Recall that in Birch, BERT input is composed from the "title" portion of TREC topics. Beyond this same input, Dai and Callan [2019b] also explored alternative query representations: the "title" field as well as the "description" field. Recall that the topic "titles" comprise short keyword phrases, whereas the topic descriptions represent well-formed natural language sentences (see Section 2.2).
Results and Analysis. Main results, in terms of nDCG@20, from Dai and Callan [2019b] on the Robust04 and test collections on ClueWeb09b (see Section 2.5) are presented in Table 8 , with figures directly copied from their paper. Just like in Birch and monoBERT, the retrieve-the-rerank strategy was used-in this case, the candidate documents were supplied by bag-of-words default ranking with the Indri search engine. 81 These results are shown in row (1) as "BOW". The top k = 100 results, with either title or description queries are reranked with BERT Base . Different aggregation techniques are compared against two baselines: SDM, row (2), refers to the sequential dependence model [Metzler and Croft, 2005] . On top of bag-of-words queries (i.e., treating all terms as independent unigrams), SDM contributes evidence from query bigrams that occur in the documents (both ordered and unordered). Previous studies have validated the empirical effectiveness of this technique, and in this context illustrates how keyword queries might take advantage of simple "structure" present in the query text (based purely on word order). As another point of comparison, the effectiveness of a simple learning to rank approach was also provided, shown in row (3) as "LTR". Improvements over LTR that are statistically significant (p < 0.05) are denoted with the † symbol.
The overall gains coming from BERT on ranking web pages (ClueWeb09b) are modest at best, and for the "title" case none of the aggregation techniques even beat the LTR baseline. Since it is unclear what strong conclusions can be drawn from the web test collections, we focus our analysis on Robust04. Comparing the different aggregation techniques, we find that the MaxP approach yields the highest effectiveness. The low effectiveness of FirstP on Robust04 is not very surprising, since it is not always the case that relevant material appears at the beginning of a news article. We see that SumP is almost as effective as MaxP, despite having the weakness that it performs no length normalization; longer documents will tend to have higher scores, thus creating a systematic bias against shorter documents.
Looking at the bag-of-words baseline, row (1), the results are generally consistent with the literature: We see that short title queries are more effective than sentence-length description queries; the drop is bigger for ClueWeb09b (web pages) than Robust04 (newswire articles). However, reranking with the descriptions as input to BERT is significantly more effective that reranking with titles, at least for Robust04. This means that BERT is able to take advantage of richer natural language descriptions of the information need.
The intriguing finding that reranking documents using BERT with respect to topic descriptions outperforms topic titles in Table 8 deserves additional exploration. Here, Dai and Callan [2019b] performed a more detailed analysis. In addition to considering the description and narrative fields from the Robust04 topics, they also explored a "keyword" version of those fields, stripped of punctuation as well as stopwords. For the narrative, they also discarded "negative logic" that may be present in the prose. For example, consider topic 697: In this topic, the second sentences in the narrative states relevance in a negative way, i.e., what makes a document not relevant. These are removed in the "negative logic removed" condition.
Results of these experiments are shown in Table 9 , where the rows show the different query conditions described above. 82 For each of the conditions, the average length of the query is provided: as expected, Table 9 : The effectiveness of SDM and BERT-MaxP using different query types on Robust04.
descriptions are longer than titles, and narratives are even longer. It is also not surprising that removing stopwords reduces the average length substantially. In these experiments, SDM (see above) is taken as a point of comparison, since it represents a simple attempt to exploit "structure" that is present in the query representations. Comparing the "title" query under SDM and the BOW results in Table 8 , we can confirm that SDM does indeed improve effectiveness.
The MaxP figures in the first two rows of Table 9 are identical to the numbers presented in Table 8 (same experimental conditions, just arranged differently). For SDM, we see that using description queries decreases effectiveness compared to the title queries, row (2a). In contrast, BERT is able to take advantage of the linguistically richer description field to improve ranking effectiveness, also row (2a). If we use only the keywords that are present in the description (only half of the terms), SDM is able "gain back" its lost effectiveness, row (2b). However, quite surprisingly, we also see from row (2b) that removing stopwords and punctuation from the description actually decreases effectiveness with BERT-MaxP! This is worth restating in another way: stopwords (that is, non-content words) contribute to ranking effectiveness in the input sequence fed to BERT for inference. These terms, by definition, do not contribute content, but rather provides the linguistic structure to help the model estimate relevance.
Looking at the narratives, which on average are over ten times longer than the title queries, we see the same general pattern. 83 SDM is not effective with long narrative queries, as it becomes "confused" by extraneous words present that are not central to the information need, row (3a). By focusing only on the keywords, SDM performs much better, but still worse than title queries, row (3b).
Removing negative logic has minimal affect on effectiveness compared to the full narrative queries, as the queries are still quite long, row (3c). For BERT-MaxP, reranking with full topic narratives beats reranking with only topic titles, but is still worse than reranking with the topic descriptions, row (3a). As is consistent with the descriptions case, retaining only keywords hurts effectiveness, demonstrating the important role that non-content words play. For BERT, removing the negative logic has negligible effect overall, just as with SDM; there doesn't seem to be sufficient evidence to draw conclusions about each model's ability to handle negations.
To further explore these findings, Dai and Callan [2019b] conducted some analyses of attention patterns in their model, along the lines of the studies mentioned in Section 3.2, although not in a systematic manner. Nevertheless, they did report a few intriguing observations: for the description query "Where are wind power installations located?", a high-scoring passage contains the sentence "There were 1,200 wind power installations in Germany." Here, the preposition in the document "in" receives the strongest attention from the term "where" in the topic description. The preposition appears in the phrase "in Germany", which precisely answers a "where" question. This represents a concrete example where non-content words play an important role in relevance matching: these are exactly the types of terms that would be discarded with exact-match techniques! Padaki et al. [2020] followed up on the work of Dai and Callan [2019b] to explore the potential of using query expansion techniques to generate better queries for BERT-based rankers. In one experiment, they scrapped Google's query reformulation suggestions based on the topic titles, which were then manually filtered to retain only those suggestions that were well-formulated natural language questions semantically similar to the original topic descriptions. While reranking using these suggestions was not as effective as reranking using the original topic descriptions, they still 83 Here, BERT is reranking results from title queries.
improved over reranking with titles only. This provided additional supporting evidence that BERT not only exploits relevance signals in well-formed natural language questions, but critically depends on them to achieve maximal effectiveness.
Are we able to make meaningful comparisons between Birch and BERT-MaxP? Given that they both present evaluation results on Robust04, there is a common point for comparison. However, there are several crucial differences that make this comparison difficult: Birch uses BERT Large whereas BERT-MaxP uses BERT Base . All things being equal, a larger (deeper) transformer model will be more effective. There are more differences: BERT-MaxP only reranks the top k = 100 results from first-stage retrieval, whereas Birch reranks the top k = 1000 hits. For this reason, computing MAP (at the standard cutoff of rank 1000) for BERT-MaxP would not yield a fair comparison to Birch. Additionally, Birch combines evidence from the original BM25 document scores, whereas BERT-MaxP does not consider scores from first-stage retrieval.
Finally, there is the issue of training. Birch entirely operates in a transfer setting, since it was fine-tuned on the MS MARCO passage retrieval test collection and TREC Microblog Track data; Robust04 data was used only to learn the sentence weight parameters. In contrast, the BERT-MaxP results come from fine-tuning directly on Robust04 data in a cross-validation setting. Obviously, in-domain training data should yield higher effectiveness, but the heuristic of constructing overlapping passages and simply assuming that they are relevant leads inevitably to noisy training examples. In contrast, Birch benefits from far more training examples from MS MARCO (albeit out of domain). It is unclear how to weigh the effects of these different training approaches.
In short, there are too many differences between Birch and BERT-MaxP to properly isolate and attribute effectiveness differences, although as a side effect of evaluating PARADE, a model we discuss in Section 3.3.4, presented experiment results that try to factor away these differences; see additional discussions there. Nevertheless, on the whole, the effectiveness of the two approaches is quite comparable: in terms of nDCG@20, 0.529 for BERT-MaxP with description, 0.533 for Birch with three sentences, MS MARCO → MB fine-tuning, which is row (4c) in Table 7 . Nevertheless, at a high level, the success of these two approaches demonstrate the robustness and simplicity of BERT-based approaches to text ranking. This also explains the rapid rise in the popularity of such models-these results are simple, effective, and easy to reproduce.
Takeaway Lessons. There are two important takeways from the work from Dai and Callan [2019b] :
• Simple maximum passage score aggregation-taking the maximum of all the passage relevance scores as the document relevance score-works well.
• BERT can exploit linguistically rich descriptions of information needs that include non-content words to estimate relevance, which is a departure from previous keyword search techniques.
The first takeaway is consistent with Birch results. Conceptually, MaxP is quite similar to the "1S" condition of Birch, where the score of the top sentence is taken as the score of the document. Birch reported at most small improvements, if any, when multiple sentences are taken into account, and no improvements beyond the top three sentences. The effectiveness of both techniques is also consistent with previous results reported in the information retrieval literature. There is a long thread of work, dating back to the 1990s, that leverages passage retrieval techniques for document ranking [Hearst and Plaunt, 1993 , Callan, 1994 , Wilkinson, 1994 , Kaszkiel and Zobel, 1997 , Clarke et al., 2000 ]-that is, aggregating passage-level evidence to estimate the relevance of a document. In fact, both the "Max" and "Sum" aggregation techniques were already explored over 25 years ago in Hearst and Plaunt [1993] and Callan [1994] , albeit the source of passage-level evidence was far less sophisticated than the transformer models of today.
Additional evidence from user studies suggest why BERT-MaxP and Birch work well: Mani et al. [2002] discovered that providing users short summaries of documents could substantially shorten the amount of time required to make relevance judgments, without any significant differences in the accuracy of the judgments compared to providing users with the full text. This finding was recently replicated and expanded upon by : the authors found that showing users only document extracts reduced both assessment time and effort in the context of a high-recall retrieval task. In a relevance feedback setting, presenting users with only isolated sentences extracts led to comparable accuracy but reduced effort compared to showing full documents . Not only from the perspective of ranking techniques, but also from the perspective of users, short, well-selected extracts serve as good proxies for entire documents for the purpose of assessing relevance. There are caveats, however: results presented below suggest that larger portions of documents need to be considered to differentiate between different grades of relevance (e.g., relevant vs. highly relevant).
Extensions. As an extension of BERT-MaxP, Wu et al. [2020b] considered whether having graded passage-level relevance judgments at training time can lead to a more effective ranking model. To answer this question, the authors annotated a corpus of Chinese news articles with passage-level cumulative gain, defined as the amount of relevant information a reader would encounter after having read a document up to a given passage. The work operationalized passages as paragraphs. The document-level cumulative gain is then, by definition, the highest passage-level cumulative gain.
Based on these human annotations, the authors made the following two observations:
• On average, highly-relevant documents are longer than other types of documents, measured both in terms of the number of passages and the number of words. • The higher the document-level cumulative gain, the more passages that need to be read by a user before the passage-level cumulative gain reaches document-level cumulative gain.
These findings suggest that whether a document is relevant can be accurately predicted from its most relevant passage-which is consistent with BERT-MaxP and Birch, as well as the user studies cited above. However, to accurately distinguish between different relevance grades (e.g., relevant vs. highly-relevant), a model might need to accumulate evidence from multiple passages, which suggests that BERT-MaxP might not be sufficient. Intuitively, the importance of observing multiple passages is related to how much relevance information accumulate across the full document.
To make use of their passage-level relevance labels, Wu et al. [2020b] proposed the Passage-level Cumulative Gain model (PCGM), which begins by applying BERT to obtain individual query-passage representations (i.e., the final representation of the [CLS] token). The sequence of query-passage representations are then aggregated with an LSTM, and the model is trained to predict each passage's gain. An embedding of the previous passage's predicted gain is concatenated to the query-passage representation to complete the model. At inference time, the gain of a document's final passage is used as the document-level gain. One can think of PCGM as a more principled approach to aggregating evidence from multiple passages than Birch, which simple takes the weighted sum of the top k passage scores. PCGM has two main advantages: the LSTM is able to model and extract signal from the sequence of passages, and, unlike both BERT-MaxP and Birch, the model is differentiable and thus amenable to end-to-end training.
The PCGM model is evaluated on two Chinese test collections. While experimental results demonstrate some increase in effectiveness over BERT-MaxP, the increase was not statistically significant. Unfortunately, the authors did not evaluate on Robust04, and thus a comparison to Birch is difficult. However, it is unclear whether the lack of significant improvements is due to the design of the model, the relatively small dataset, or some issue with the underlying observations about passage-level gains. Nevertheless, the intuitions of Wu et al. [2020b] in recognizing the need to aggregate passage representations does appear to be valid, and in Section 3.3.4 we discuss additional work along these lines that is able to significantly outperform BERT-MaxP.
Just as in applications of BERT to classification tasks in NLP (see Section 3. Figure 10 : The architecture of CEDR, which comprises two main relevance signals: the [CLS] representation and the signals extracted from the similarity matrix computed from contextual embeddings of the texts. This illustration contains a number of simplifications in order to clearly convey the model's high-level design.
• As with pre-BERT interaction-based neural ranking models, CEDR constructs similarity matrices between terms from the query and terms from the candidate text. CEDR then concatenates the contextual embeddings for terms from the candidate text in each chunk to form the sequence of contextual embeddings for the entire text.
In other words, CEDR performs chunk-by-chunk inference over long texts, and then assembles relevance signals from each chunk (i.e., the [CLS] representations and contextual embeddings).
From the scientific perspective, MacAvaney et al.
[2019a] investigated whether BERT's contextual embeddings outperform static embeddings when used in a pre-BERT neural ranking model and whether they are complementary to the more commonly used [CLS] representation. They hypothesized that since interaction-based models rely on the ability of the underlying embeddings to capture semantic term matches, using richer contextual embeddings to construct the similarity matrix should improve the effectiveness of interaction-based neural ranking models.
Specifically, CEDR uses one of three neural ranking models as a "base": DRMM , KNRM [Xiong et al., 2017] , and PACRR [Hui et al., 2017] . Instead of static embeddings (e.g., from GloVe), the embeddings that feed these models now come from BERT. In addition, the aggregate [CLS] representation from BERT is concatenated with the other signals consumed by the feedforward network of each base model. Thus, query-document relevance scores are derived from two main sources: the [CLS] token (as in monoBERT, Birch, and BERT-MaxP) and from signals derived from query-document term similarities (as in pre-BERT interaction-based models). This overall design is illustrated in Figure 10 . The model is more complex than can be accurately captured in a diagram, and thus we only attempt to highlight high-level aspects of the design.
To handle inputs longer than 512 tokens, CEDR splits documents into smaller chunks, as evenly as possible, such that the length of each input sequence (complete with the query and special delimiter tokens) is no more than the 512 token maximum. BERT processes each chunk independently and the output from each chunk is retained. Once all of a document's chunks have been processed, CEDR creates a document-level [CLS] representation by averaging the [CLS] representations from each chunk (i.e., average pooling). The document-level [CLS] representation is then concatenated with the relevance signals that are fed to the underlying interaction-based neural ranking model. Unlike in monoBERT, Birch, and BERT-MaxP, which discard the contextual embeddings of the query and documents, CEDR concatenates the contextual embeddings of the document terms from each chunk to form the complete list of contextual term embeddings for the entire document. Similarity matrices are then constructed by computing the cosine similarity between each document term embedding and the query term embeddings from the first document chunk. Note that BERT is incorporated into interaction-based neural ranking models in a way the retains the differentiability of the overall ranking model. This allows end-to-end training with relevance judgments and provides the solution to the length limitations of BERT.
Given that the input size in a transformer encoder is equal to its output size, each layer in BERT can be viewed as producing some (intermediate) representation and similarity matrix signals, CEDR produces a final document relevance score by using the same series of fully-connected layers that is used by the underlying base neural ranking model. In more detail:
• CEDR-DRMM uses a fully-connected layer with five output nodes and a ReLU non-linearity followed by a fully-connected layer with a single output node.
• CEDR-KNRM uses one fully-connected layer with a single output node.
• CEDR-PACRR uses two fully-connected layers with 32 output nodes and ReLU non-linearities followed by a fully-connected layer with a single output node.
All variants are trained using a pairwise hinge loss and initialized with BERT Base . The final querydocument relevance scores are then used to rerank a list of candidate documents.
As a baseline model for comparison, MacAvaney et al. [2019a] proposed what they called "Vanilla BERT", which is an ablated version of CEDR that uses only the signals from the [CLS] representations. Specifically, documents are split into chunks in exactly the same way as the full CEDR model and the [CLS] representations from each chunk are averaged before feeding a standard relevance classifier (as in monoBERT, Birch, and BERT-MaxP). This model quantifies the impact on effectiveness of the query-document term interactions.
Results and Analysis. CEDR was evaluated using the Robust04 and a non-standard combination of datasets from the TREC 2012-2014 Web Tracks that we simply denote as "Web" (see Section 2.5 and the original paper for details). Results in terms of nDCG@20 are shown in Table 10 , with figures copied directly from MacAvaney et al. [2019a] . CEDR was deployed as a reranker over BM25 results from Anserini, the same as Birch. However, since CEDR only reranks the top k = 100 hits (as opposed to k = 1000 hits in Birch), the authors did not report MAP. Nevertheless, since nDCG@20 is an early-precision metric, the scores can be meaningfully compared. 84 Copying the conventions used by the authors, the prefix before each result in brackets denotes significant improvements over BM25, Vanilla BERT, the corresponding model trained with GloVe embeddings, and the corresponding Non-CEDR model (i.e., excluding [CLS] signals), based on paired t-tests (p < 0.05).
In Table 10 , each block of rows focuses on a particular "base" interaction-based neural ranking model, where the rows with the "CEDR-" prefix denote the incorporation of the [CLS] representations. The "Input Representation" column indicates whether static GloVe embeddings [Pennington et al., 2014] or BERT's contextual embeddings are used. When using contextual embeddings, the original versions from BERT may be used or the embeddings may be fine-tuned on the ranking task along with the underlying neural ranking model. When BERT is fine-tuned on the ranking task, a Vanilla BERT model is first fine-tuned before training the underlying neural ranking model. That is, BERT is first fine-tuned in the Vanilla BERT configuration for relevance classification, and then it is fine-tuned further in conjunction with a particular interaction-based neural ranking model.
Let us examine these results. First, consider whether contextual embeddings improve over static GloVe embeddings: the answer is clearly yes. Even without fine-tuning on the ranking task, BERT embeddings are slightly more effective than GloVe embeddings across all models and datasets. This
Web Method Input Representation nDCG@20 nDCG@20
( is shown in the (b) rows vs. the (a) rows. Fine-tuning BERT yields additional large improvements for most configurations, with the exception of DRMM on the Web data. This is shown in the (c) rows vs. the (b) rows.
Next, consider the effectiveness of using only contextual embeddings in an interaction-based neural ranking model compared with the effectiveness of using only the [CLS] representation, represented by Vanilla BERT in row (2). When used with contextual embeddings, the PACRR and KNRM models perform substantially better than Vanilla BERT; see the (c) rows vs. row (2). DRMM does not appear to be effective in this configuration, however. This may be caused by the fact that DRMM's histograms are not differentiable, which means that BERT is fine-tuned using only the relevance classification task (i.e., BERT's weights are updated when Vanilla BERT is first fine-tuned, but BERT's weights are not updated when CEDR-DRMM's is fine-tuned after). Nevertheless, there is some reason to suspect that the effectiveness of Vanilla BERT is under-reported, perhaps due to some training issue, because an equivalent approach by is much more effective. A natural question that arises is how CEDR compares against Birch (Section 3.3.1) and BERT-MaxP (Section 3.3.2), the two other contemporaneous models in the development of BERT for ranking full documents. Fortunately, all three models were evaluated on Robust04 and nDCG@20 was reported for those experiments, which provides a common reference point. Table 11 summarizes the best configuration of each model, and we can see that the effectiveness appears to be quite comparable. This point has already been mentioned in Section 3.3.2 but is worth repeating: it is quite remarkable that three ranking models with different designs, by three different research groups with experiments conducted on independent implementations, all produce similar results. This provides robust evidence that BERT does really "work" for text ranking.
The connection between Birch and BERT-MaxP has already been discussed in the previous section, but both models are quite different from CEDR, which has its design more firmly rooted in pre-BERT interaction-based neural ranking models. 
PARADE , which stands for Passage Representation Aggregation for Document Reranking, is a direct descendant of CEDR that also incorporates lessons learned from Birch and BERT-MaxP. The key insight of PARADE, building on CEDR, is to aggregate the representations of passages from a long text rather than aggregating the scores of individual passages, as in Birch and BERT-MaxP. This yields a differentiable model that can consider multiple passages in unison. This design also unifies training and inference, as opposed to distinct training and inference approaches in Birch and BERT-MaxP. These are the same two advantages of CEDR discussed above. However, PARADE abandons CEDR's connection to pre-BERT neural ranking models by discarding the explicit term-interaction similarity matrices. The result is ranking model that is both simpler than CEDR and more effective.
More precisely, PARADE is a family of models that splits a long text into passages and performs representation aggregation on the [CLS] representation from each passage. Specifically, PARADE splits a long text into a fixed number of fixed-length passages. When texts contain fewer passages, the passages are padded and masked out during representation aggregation. When texts contain more passages, the first and last passages are always retained, and the remaining passages are randomly sampled. Consecutive passages partially overlap to minimize the chance of separating relevant information from its context.
A passage representation p cls i is computed for each passage P i given a query q using BERT Base : • PARADE Attn computes a weighted average of the passage representations by using a feedforward network to produce an attention weight for each passage. That is, Figure 11 .
Note that the first three approaches treat each dimension of the passage representation as an independent feature. That is, pooling is performed across passage representations. In all cases, the final document representation d cls is fed to a fully-connected layer with two output nodes that then feeds a softmax to produce the final relevance score.
PARADE's hierarchical approach follows a line of prior work on hierarchically modeling natural language text, which, to our knowledge, began in the context of deep learning with Hierarchical Attention Networks (HANs) for document classification [Yang et al., 2016] . The architecture uses two layers of RNNs to model text at the world level and at the sentence level. extended this basic strategy to three levels (paragraphs, sentences, and words) and applied the resulting model to semantic text matching of long pieces of text. PARADE's approach is most similar to that by Liu and Lapata [2019] and , who proposed a hierarchical transformer for document classification.
Results and Analysis. evaluated the PARADE models on the Robust04 and Gov2 datasets using both title (keyword) and description queries (see Section 2.5). Each PARADE model is built on top of a BERT Base model, which was first fine-tuned on the MSMARCO passage ranking task (more on this in Section 3.5.5). The entire model was then fine-tuned on the target test collection using (4), a model that has already been fine-tuned on MS MARCO is used as a starting point to additional finetuning on collection-specific relevance judgments. For the PARADE models, statistically significant improvements over Birch (MS), Birch (MS→MB) and BERT-MaxP (MS) are indicated with the symbols † , ‡ and § , respectively, based on paired t-test (p < 0.01).
Results suggest that ranking effectiveness increases with more sophisticated representation aggregation approaches overall. PARADE , which performs aggregation using transformer encoders, is consistently the most effective model across metrics, query types, and datasets; PARADE Avg is consistently the least effective. PARADE Max and PARADE Attn perform similarly, with PARADE Attn slightly outperforming PARADE Max in most configurations. Furthermore, PARADE consistently outperforms the BERT-MaxP and Birch score aggregation approaches, but the results are mixed for the other PARADE variants. PARADE Attn performs slightly worse than Birch on title queries and similarly to BERT-MaxP, with higher effectiveness on Robust04 and lower effectiveness on Gov2. PARADE Attn outperforms both score aggregation approaches on description queries.
As an additional contrastive experiment, investigated replacing BERT Base with BERT Large , also shown in Table 12 in row (6). Comparing row (5d) with row (6) suggests that PARADE does not benefit from using the larger BERT Large model, though the authors only provided BERT Large results on a subset of the datasets. This is an interesting finding, as experiments in the context of monoBERT (Section 3.2) clearly show the benefit of a larger BERT model. It could be the case that the evidence aggregation is compensating for the weaker models at the level of individual passages; this is an interesting hypothesis that would benefit from additional investigation.
Note that although PARADE Avg is the least effective technique, it still represents a large improvement over bag-of-word baselines. Setting aside implementation details, PARADE Avg is equivalent to the Vanilla BERT condition in CEDR. Yet, PARADE Avg alone achieves effectiveness that is on par with the complete CEDR models (with additional signals from the similarity matrices); this observation suggests that the Vanilla BERT condition under-performs its true effectiveness.
In addition to providing a point of comparison for PARADE, these experiments also shed additional insight about differences between Birch and BERT-MaxP, a question we broached in Section 3.3.2 but did not completely resolve. Here, it is worth spending some time discussing these results, independent of PARADE. At a high level, these findings are consistent with the original papers, and thus these experiments confirm that the results reported by the original authors are reproducible.
Interestingly, Birch performs worse on description queries than on title queries on both test collections, which appears inconsistent with the findings of Dai and Callan [2019b] (see Section 3.3.2). However, this may be due to the mismatch between the queries at training time and at inference time; in row (3b), Birch is ultimately fine-tuned with microblog data, comprising keyword queries. The results in row (3a) show that fine-tuning with only MS MARCO data is not effective for relevance classification on sentences, which is consistent with the findings of [Akkalyoncu (who observed limited gains). It is worth noting that Birch uses a BERT Large model here (as in the original paper), whereas BERT-MaxP and the PARADE variants use the smaller Base model. Confirming the findings reported by Dai and Callan [2019b] , the effectiveness of BERT-MaxP increases when moving from Robust04 title queries to description queries. This successfully reproduces the result showing that BERT benefits from well-formed natural language descriptions of the information needs. However, on Gov2 the description queries perform worse.
Overall, compared with results reported in the original papers, BERT-MaxP's effectiveness is higher and Birch's is lower. The increase in the effectiveness of BERT-MaxP is likely from the fact that, like the PARADE models, it was first fine-tuned on the MS MARCO passage retrieval test collection before further fine-tuning on the target dataset (see more discussion in Section 3.5.5). The decrease in effectiveness observed in Birch is likely due to the difference in experimental setups: k = 100 documents from BM25 were reranked rather than k = 1000 documents from BM25 + RM3, as in the original paper.
Takeaway Lessons. We see two main takeaways from PARADE: First, aggregating passage representations is more effective than aggregating passages scores. Passage representations are far richer, and by the time a passage score has been computed, a lot of the signal has been "lost". Second, chunking a long text and performing chunk-level inference seems like a tenable strategy to addressing the length restrictions of BERT, as an alternative to solutions that try to directly increase the maximum length of input sequences to BERT, such as Reformer [Kitaev et al., 2020] and Longformer . The key lies in proper aggregation of representations that emerge from inference passes over the individual chunks. Pooling, particularly max pooling, is a simple and effective technique, but using another transformer to aggregate the individual representations is even more effective, showing that there are rich signals that can be extracted from the sequence of chunk-level representations. This hierarchical approach to relevance modeling retains the important model property of differentiability, enabling the unification of training and inference.
The applications of BERT to text ranking that we have covered so far operate as rerankers in a retrieve-and-rerank setup, which as we have noted dates at least back to at least the 1960s [Simmons, 1965] . An obvious extension of this design is to incorporate multiple reranking stages as part of a multi-stage ranking architecture, as shown in Figure 12 . That is, following candidate generation or first-stage retrieval, instead of having just a single reranker, a system could have an arbitrary number of reranking stages, where the output of each reranker feeds the input to the next. This basic design goes by a few other names as well: reranking pipelines, ranking cascades, or "telescoping".
We formalize the design as follows: a multi-stage ranking architecture comprises N reranking stages, denoted H 1 to H N . We refer to the candidate generation stage (i.e., first-stage retrieval) as H 0 , which retrieves k 0 texts from an inverted index. Each stage H n , n ∈ {1, . . . N } receives a ranked list R n−1 comprising k n−1 candidates from the previous stage. Each stage, in turn, provides a ranked list R n comprising k n candidates to the subsequent stage, with the requirement that k n ≤ k n−1 . 85 The Figure 12 : A retrieve-and-rerank setup (top) is the simplest instantiation of a multi-stage ranking architecture (bottom). In multi-stage ranking, the candidate generation stage is followed by more than one reranking stages.
ranked list generated by the final stage H N is the output of the multi-stage ranking architecture. This description intentionally leaves unspecified the implementation of each reranking stage, which could be anything ranging from decisions made based on the value of a single hand-crafted feature (for example, a decision stump) to a sophisticated machine-learned model (for example, based on BERT). Furthermore, each stage could decide how to take advantage of scores from the previous stage-one common design is that scores from each stage are additive-or a reranker can decide to completely ignore previous scores.
One practical motivation for the development of multi-stage ranking is to better balance tradeoffs between effectiveness (quality of the ranked lists) and efficiency (retrieval latency). Users, of course, demand systems that are both "good" and "fast", but in general, there is a natural tradeoff between these two desirable characteristics. Multi-stage ranking evolved in the context of learning to rank (see Section 1.2.3): Compared to unigram features (i.e., on individual terms) such as BM25 scores, many n-gram features are better signals of relevance, but also more computationally expensive to compute, in both time and space. As a simple example, one helpful feature is the count of query n-grams that occur in a text (that is, the system checks that matching query terms are contiguous). This is typically accomplished by storing the positions of terms in the text (which consumes space) and intersecting lists of term positions (within individual documents) to determine whether the terms appear contiguously (which takes time). Thus, we see a common tradeoff between feature cost and quality, and more generally, between effectiveness and efficiency.
Thus, a ranking model (e.g., learning to rank) that takes advantage of "expensive" features will often be slow, since inference must be performed on every candidate document. As ranking latency increases linearly with respect to the number of candidates, it can be managed by varying the depth of first-stage retrieval, much like the experiments presented in Section 3.2 in the context of monoBERT. However, it is desirable that the candidate pool contains as many relevant texts as possible (i.e., have high recall), to maximize the opportunities for a reranker to identify relevant texts; obviously, rerankers are useless if there are no relevant texts in the output of first-stage retrieval to act on. Thus, designers of production real-world systems are faced with an effectiveness/efficiency tradeoff.
The intuition behind the multi-stage design is to exploit expensive features only when necessary: earlier stages in the reranking pipeline can use "cheap" features to discard candidates that are easy to distinguish as not relevant; "expensive" features can then be brought to bear after the "easy" cases have been discarded. Latency can be managed because increasingly expensive features are computed on increasingly fewer candidates. Furthermore, reranking pipelines can exploit "early exits" that bypass later stages if the results are "good enough" [Cambazoglu et al., 2010] . In general, the multi-stage design provides system designers with tools to balance effectiveness and efficiency, often leading to systems that are both "good" and "fast". 86
The development of this idea in modern times has an interesting history. It had been informally known by many in the information retrieval community since at least the mid-2000s that Microsoft's Bing search engine adopted a multi-stage design; for one, it was the most plausible approach for deploying the learning to rank models they were developing [Burges et al., 2005] . However, the earliest "official" public acknowledgment appears to be in a SIGIR 2010 Industry Track keynote by Jan Pedersen, whose presentation included a slide that explicitly showed this multi-stage architecture. Bing named these stages "L0" through "L4", with "L0" being "Boolean logic" (understood to be conjunctive query processing with the ANDing of query terms), "L1" being "IR score" (understood to be BM25), and "L2/L3/L4" being machine-learned models. Earlier that year, a team of authors from Yahoo! [Cambazoglu et al., 2010 ] described a multi-stage ranking architecture in the form of with additive ensembles (the score of each stage is added to the score of the previous stages). However, the paper did not establish a clear connection to production systems.
In the academic literature, Matveeva et al. [2006] described the first known instance of multi-stage ranking ("nested" rankers, as the authors called it). The term "telescoping" was used to describe the pruning process where candidates were discarded between stages. Interestingly, the paper was motivated by high-accuracy retrieval and did not discuss the implications of their techniques on system latency. Furthermore, while four of the five co-authors were affiliated with Bing, the paper provided no indications of or connections to the web search engine's design. One of the earliest academic papers to include efficiency objectives into learning to rank was by Wang et al. [2010] , who explicitly modeled feature costs in a framework to jointly optimize effectiveness and efficiency; cf. [Xu et al., 2012] . In a follow up, Wang et al. [2011] proposed a boosting algorithm for learning ranking cascades to directly optimize this quality/speed tradeoff. Within the academic literature, this is the first instance we are aware of that describes learning the stages in a multi-stage ranking architecture. Nevertheless, it is clear that industry led the way in explorations of this design, but since there is paucity of published material, we have no public record of when various important innovations occurred.
Since the early 2010s, multi-stage ranking architectures have received substantial interest in the academic literature [Tonellotto et al., 2013 , Asadi and Lin, 2013 , Capannini et al., 2016 , Clarke et al., 2016 , Chen et al., 2017b , Mackenzie et al., 2018 as well as industry. Among publicly documented deployments at scale, beyond Bing, it appears that Alibaba's e-commerce search engine adopts this same architecture as well .
The first application of transformers in a multi-stage ranking architecture was described by Nogueira et al. [2019a] as a solution for mitigating the quadratic computational costs associated with a particular implementation of pairwise ranking, as we explain below.
Recall that monoBERT (Section 3.2) turns ranking into a relevance classification problem, where we sort texts by P (Relevant = 1|d i , q) given a query q and candidates {d i }. In the terminology of learning to rank, this is considered a "pointwise" approach since each text is considered in isolation [Liu, 2009 , Li, 2011 ]. An alternative is the "pairwise" approach, which focuses on comparisons between pairs of documents. Intuitively, pairwise ranking has the advantage of harnessing information present in other candidate texts to decide if a text is relevant to a given query. The "duoBERT" model proposed by Nogueira et al. [2019a] operationalizes exactly this intuition. In this ranking model, BERT is trained to estimate the following:
where d i d j is a commonly adopted notation for stating that d i is more relevant than d j (with respect to the query q).
Before going into details, there are conceptually two challenges to realizing this ranking strategy: 87 86 Note an important caveat here is the assumption that users only desire a few relevant documents, as is typical in web search and operationalized in terms of early-precision metrics. Multi-stage architectures are not as useful if users desire high recall, as in the applications we discuss in Section 3.7. 87 A possible point of confusion here: in learning to rank, "pairwise" and "pointwise" refer to the form of the loss, not the form of the inference mechanism. For example, RankNet [Burges et al., 2005] is trained in a pairwise 1. The result of model inferences comprises a set of pairwise comparisons between candidate texts. Evidence from these pairs still need to be aggregated to produce a final ranked list. 2. One simple implementation is to compare each candidate to every other candidate (e.g., from first-stage retrieval), and thus the computational costs increase quadratically with the size of the candidate set. Recall monoBERT experiments have shown that effectiveness increases with the size of the candidates set (see Section 3.2): thus, there emerges an effectiveness/efficiency tradeoff. We need some mechanism to control this tradeoff. Nogueira et al. [2019a] proposed a number of evidence aggregation strategies (described below), but a multi-stage ranking architecture provides a nice solution to the second challenge. In summary, in a multi-stage design, a relevance classifier can be used to select a smaller set of candidates from first-stage retrieval that will be fed to the pairwise reranker.
The duoBERT model is trained to estimate p i,j , the probability that d i d j , i.e., candidate d i is more relevant than d j . It takes as input a sequence comprised of a query and two texts: token is used as input to a fully-connected layer to obtain the probability p i,j . For k candidates, |k| × (|k| − 1) probabilities are computed. The model is trained end-to-end with the following loss:
Note in the equation above that candidates d i and d j are never both relevant or not relevant. Since this loss function considers pairs of candidate texts, it can be characterized as belonging to the family of pairwise learning to rank approaches [Liu, 2009 , Li, 2011 . For additional training details, including hyperparameter settings, we refer the reader to the original paper.
At inference time, the pairwise scores p i,j are aggregated so that each document receives a single score s i . Nogueira et al. [2019a] investigated five different aggregation methods, four of which we discuss below:
MAX :
MIN :
SUM :
BINARY :
where J i = {0 ≤ j < |D|, j = i} and m is the number of samples drawn without replacement from the set J i . The SUM method measures the pairwise agreement that candidate d i is more relevant than the rest of the candidates {d j } j =i . The BINARY method is inspired by the Condorcet method [Montague and Aslam, 2002] , which provides a strong aggregation baseline [Cormack et al., 2009] . The MIN (MAX) method measures the relevance of d i only against its strongest (weakest) "competitor". The final ranked list (for evaluation) is obtained by reranking the candidates according to their scores s i .
Results on the MS MARCO passage retrieval test collection are shown in Table 13 , organized in the same manner as Table 5 ; the experimental conditions are directly comparable. Row (1) reports the manner (i.e., loss is computed with respect to pairs of texts), but inference (i.e., at query time) is performed on individual texts. In duoBERT, both training and inference are performed on pairs of texts. effectiveness of Anserini's initial candidates using BM25 scoring. In row (2), BM25 results reranked with monoBERT with BERT Large (k 0 = 1000) are shown, which is exactly the same as row (6) in Table 5 . Rows (3)-(5) report results from reranking the top 50 results from the output of monoBERT (i.e., k 1 = 50) using the various aggregation techniques presented above. Effectiveness in terms of the official metric MRR@10 is reported on the publicly available development set for all aggregation methods (i.e., duoBERT using BERT Large ), but Nogueira et al. [2019a] only submitted results from the SUM condition for evaluation on the test set. It is clear that MAX aggregation is not as effective as the other three techniques, but the difference between MIN, SUM, and BINARY are all negligible.
In all the experimental conditions above, duoBERT considers the top 50 candidates from monoBERT output (i.e., k 1 = 50), and thus requires an additional 50 × 49 BERT inferences to compute the final ranking (the time required for aggregation is negligible). Nogueira et al. [2019a] argued that total number of BERT inferences is an easy yet reasonably accurate way to capture overall query latency because monoBERT and duoBERT models have similar inference latencies. From this perspective, the multi-stage design with duoBERT does not seem compelling, since it costs 50 × 49 = 2450 additional inferences for only half a point gain in MRR@10 on the test set. Since monoBERT with k 0 = 1000 requires 1000 BERT inferences, this is a 3.5× increase in latency. The tradeoffs, however, do become much more compelling if we consider that a two-stage design provides a bigger design space for selecting a desirable operating point to balance effectiveness and efficiency. With a single reranking stage (monoBERT), the only choice is to vary the k 0 parameter, but with two rerankers, it is possible to simultaneously tune k 0 and k 1 . These tradeoff curves are shown in Figure 13 , with duoBERT SUM for aggregation. The gray line shows effectiveness with different values of k 0 for monoBERT (same as the values in Figure 7 ). The other lines show settings of k 1 ∈ {10, 30, 50}, and with each k 1 setting, points in the tradeoff curve are traced by k 0 = {50, 100, 200, 500, 1000}. In the two-stage configuration, the number of inferences per query is calculated as k 0 + k 1 (k 1 − 1).
Hypothetical vertical lines intersecting with each curve denote the best effectiveness that can be achieved with a particular computational budget: these results show that if a system is willing to expend more than couple of hundred BERT inferences, then a two-stage configuration is more effective overall. That is, rather than simply increasing the reranking depth of monoBERT, it appears better to reallocate some of the effort to a pairwise approach that examines pairs of candidate texts. The Pareto frontier in the effectiveness-efficiency tradeoff space is shown in Figure 13 as the dotted black line. For each point on the frontier, there exists no other setting that achieves both higher MRR@10 while requiring fewer inferences.
Takeaway Lessons. Multi-stage ranking architectures represent a straightforward generalization of the retrieve-then-rerank approach adopted in monoBERT. Introducing multiple rerankers in a pipeline greatly expands the possible operating points of an end-to-end system in the effectiveness/efficiency tradeoff space, potentially leading to settings that are both better and faster than what can be achieved with a single-stage reranker. On potential downside, however, is that multi-stage pipelines introduce additional "tuning knobs" that need to be properly adjusted to strike a desired tradeoff. In the monoBERT/duoBERT design, these parameter settings (k 0 , k 1 ) are difficult to learn as the pipeline is not differentiable end-to-end. Thus, the impact of different parameter settings must be empirically determined from a test collection. Figure 7 . For monoBERT + duoBERT SUM , each curve plots a different k 1 , and points in each curve correspond to k 0 = {50, 100, 200, 500, 1000}; the number of inferences per query is calculated as k 0 + k 1 (k 1 − 1). The Pareto frontier is shown as the dotted black line.
Multi-stage ranking pipelines exploit faster (and possibly less effective) models in earlier stages to discard likely non-relevant documents so there are fewer candidates under consideration by more expensive models in later stages. In the case of the monoBERT/duoBERT architecture described above, the primary goal was to make a more inference-heavy model (i.e., duoBERT) more practical. That is, the goal was to improve the quality of monoBERT ranking while maintaining acceptable effectiveness/efficiency tradeoffs. However, the architecture isn't particularly useful if we desire a system that is even faster (but perhaps less effective) than the baseline monoBERT design. In this case, one possibility is to use a standard telescoping pipeline that potentially include pre-BERT neural ranking methods as suggested by Matsubara et al. [2020] . Given monoBERT as a starting point, another obvious solution is to leverage the large body of research on model pruning and compression, which is not specific to text ranking or even natural language processing. In Section 3.6.1, we cover knowledge distillation, thread in this broad space. Here, we discuss a solution that shares similar motivations, but is clearly inspired by multi-stage ranking architectures. Soldaini and Moschitti [2020] began with the observation that a model like monoBERT is already like a multi-stage ranking architecture if we consider each layer of the transformer encoder as a separate ranking stage. In the monoBERT design, inference is applied to all input texts (typically, k 0 = 1000). This seems like a "waste" if the model somehow already knew that a particular text was not likely to be relevant, partway through the layers. Therefore, a sketch of the solution might look like the following: start with a pool of candidate texts, apply inference on the entire batch using the first few layers, discard the least promising candidates, continue inference with the next few layers, discard the least promising candidates, and so on, until the end, when only the most promising candidates have made it all the way through the layers. With cascade transformers, Soldaini and Moschitti [2020] did exactly this.
More formally, with cascade transformers, intermediate classification decision points (which we'll call "early exits" for reasons that will become clear in a bit) are built in at layers j = λ 0 + λ 1 · (i − 1), ∀i ∈ {1, 2, . . .}, where λ 0 , λ 1 ∈ N are hyperparameters. Specifically, Soldaini and Moschitti [2020] build on the base version of RoBERTa [Liu et al., 2019c] , which has 12 layers; they used a setting of λ 0 = 4 The α parameter controls the proportion of candidates discarded at each stage of the ranking pipeline.
and λ 1 = 2, which yields five rerankers, with decision points at layers 4, 6, 8, 10, and 12. 88 The rationale for skipping the first λ 0 layers is that relevance classification effectiveness is too poor for the model to be useful; this is observation is consistent with findings across many NLP tasks [Houlsby et al., 2019 , Lee et al., 2019a , Xin et al., 2020 . The [CLS] vector representation at each of the j layers (i.e., each of the cascade rerankers) is then fed to a fully-connected classification layer that computes the probability of relevance for the text; this remains a pointwise relevance classification design. At inference time, at each of the j layers, the model will score P candidate documents and retain only the top (1 − α) · P scoring candidates, where α ∈ [0 . . . 1] is a hyperparameter, typically between 0.3 and 0.5. That is, α · P candidates are discarded at each stage.
In practice, neural network inference is typically conducted on GPUs in batches. Soldaini and Moschitti [2020] worked through a concrete example of how this these settings play out in practice: Consider a setting of α = 0.3 with a batch size b = 128. With the five cascade reranker design described above, after layer 4, the size of the batch is reduced to 90, i.e., 0.3 · 128 = 38 candidates are discarded after the first classifier. At layer 6, after the second classification, 27 additional candidates are discarded, with only 63 remaining. At the end, only 31 candidates are left. Thus, cascade transformers have the effect of reducing the average batch size, which increases throughput on GPUs compared to a monolithic design, where inference must be applied to all input instances. In the example above, suppose that based on a particular hardware configuration we can process a maximum batch size of 84 using a monolithic model. With cascade transformers, we can instead process batches of 128 instances within the same memory constraints, since (4·128+2·90+2·63+2·44+2·28)/12 = 80.2 < 84. This represents a throughput increase of 52%. The cascade transformer architecture requires training of all the classifiers at each of the individual rerankers (i.e., early exit points). The authors described a procedure wherein for every mini-batch, one of the rerankers is sampled (including the final output ranker): its loss against the target labels is computed and back-propagated throughout the entire model, down to the embedding layers. This simple uniform sampling strategy was found to be more effective than alternative techniques such as round-robin selection and biasing the early rerankers.
Soldaini and Moschitti [2020] evaluated their cascade transformer design on the answer selection task, where the goal is to select from a pool of candidate sentences the ones that contain the answer to a given question. This is essentially a text ranking task on sentences, where the ranked output provides the input to downstream modules that identify answer spans. The authors report results on multiple answer selection datasets, but here we focus on two: Answer Sentence Natural Questions (ASNQ) Garg et al. [2020] , which is a large dataset constructed by extracting sentence candidates from the Google Natural Question (NQ) dataset [Kwiatkowski et al., 2019] , and General Purpose Dataset (GPD), which is a proprietary dataset with questions submitted to Amazon Alexa and answers annotated by humans. In both cases, the dataset itself includes the candidates to be reranked (i.e., first-stage retrieval is fixed and part of the test collection itself).
Results copied from the authors' paper are shown in Table 14 . The baseline is TANDA BASE [Garg et al., 2020] , which is monoBERT with a multi-stage fine-tuning procedure that uses multiple datasets (see Section 3.5.5); this is shown in row (1). For each dataset, effectiveness results in terms of standard metrics are shown; the final column denotes an analytically computed cost reduction per batch. The cascade transformer architecture is denoted CT, in row group (2). In row (2a), with α = 0.0, all candidate sentences are scored using all layers of the model (i.e., no candidates are discarded). This model performs slightly better than the baseline, and these gains can be attributed to the training of the intermediate classification layers, since the rest of the CT architecture is exactly the same of the TANDA baseline. Rows (2b), (2c), and (2d) report effectiveness with different α settings. On the ASNQ dataset, CT with α = 0.5 is able to decrease inference cost per batch by around half with a small decrease in effectiveness. On the GPD dataset, inference cost can be reduced by 37% (α = 0.3) with a similarly modest decrease in effectiveness. These experiments clearly demonstrate that cascade transformers provide a way for system designers to control effectiveness/efficiency tradeoffs in multi-stage ranking architectures. As with the monoBERT/duoBERT design, the actual operating point depends on a number of considerations, but the main takeaway is that these designs provide the knobs for system designs to express their desired tradeoffs.
At the intersection of model design and the practical realities of GPU-based inference, Soldaini and Moschitti [2020] discuss a point that is worth repeating here. In their design, a fixed α is crucial to obtaining the performance gains observed, although in theory one could devise a number of other approaches to pruning. For example, candidates can be discarded based on a score threshold (that is, discard all candidates with score below a given threshold). Alternatively, it may even be possible to separately learn a lightweight classifier that dynamically decides the candidates to discard. The challenge with these alternatives, however, is that it becomes difficult to a priori determine batch sizes, and thus efficiently exploit GPU resources (which depend critically on regular computations).
It is worth noting that cascade transformers were designed to rank candidate sentences in a question answering task, and cannot be directly applied to document ranking, even with relatively simple architectures like Birch and BERT-MaxP. There is the practical problem of packing sentences (from Birch) or passages (from BERT-MaxP) into batches for GPU processing. As we can see from the discussion above, cascade transformers derive their throughput gains from the ability to more densely pack instances into the same batch for efficient inference. However, for document ranking, it is important to distinguish between scores of segments both within documents as well as across documents. The simple filtering decision in terms of α cannot preserve both relationships at the same time if segments from multiple documents are mixed together, but since documents have variable numbers of sentences or passages, strictly segregating batches by document will reduce the regularity of the computations and hence the overall efficiency. To our knowledge, these issues have not been tackled, and cascade transformers have not been extended for ranking texts that are longer than BERT's 512 token length limit. However, such extensions would be interesting future work.
To gain a better understanding of cascade transformers, it is helpful to situate this work within the broader context of other research in NLP. The insight that not all layers of BERT are necessary for effectively performing a task (e.g., classification) was shared independently and contemporaneously by a number of different research teams. While Soldaini and Moschitti [2020] operationalized this idea for text ranking in cascade transformers, other researchers applied the same intuition for other natural language texts. For example, DeeBERT [Xin et al., 2020] builds early exit "off ramps" in BERT to accelerate inference for test instances based on an entropy threshold; the work of Schwartz et al. [2020] implements the same idea with only minor difference in details. Quite amazingly, these two papers, along with the work of Soldaini and Moschitti, were all published at the same conference, ACL 2020! Although this remarkable coincidence suggests early exit was an idea "whose time had come", it is important to recognize that, in truth, the idea had been around for a while-just not in the context of neural networks. In 2010, Cambazoglu et al. [2010] proposed early exits in additive ensembles for ranking, but in the context gradient-boosted decision trees, which exhibit the same regular, repeating structure (at the "block" level) as transformer layers. Of course, BERT and transformers provide a fresh context that opens up new design choices, but many of the lessons and ideas from previous work remain applicable.
A final concluding thought before moving on: the above discussion suggests that the distinction between monolithic ranking models and multi-stage ranking is not clear cut. For example, is the cascade transformer a multi-stage ranking pipeline or a monolithic ranker with early exits? Both seem apt descriptions, depending on one's perspective. However, the monoBERT/duoBERT combination can only be described as multi-stage ranking, since the two rerankers are quite distinct. Perhaps the distinction lies in the "end-to-end" differentiability of the model (and hence how it is trained)? But yet, differentiability stops at the initial candidate generation stage since all the architectures discussed in this section still rely on keyword search. Continuous dense representations for ranking, which we cover in Section 4, aim to address this very point, and with the introduction of approximate nearest neighbor search as a replacement for keyword search with inverted indexes, these distinctions become even more muddled. Indeed, the relationship between these various designs remains an open question and the focus of much ongoing research activity. However, we defer high-level discussion of these issues to Section 5.
Takeaway Lessons. Cascade transformers represent another example of a multi-stage ranking pipeline. Compared to the monoBERT/duoBERT design, the approach is very different, which illustrates the versatility of the overall architecture. Researchers have only begun to explore this design space, and we expect more interesting future work to emerge.
In the text ranking scenarios we consider in this survey, the assumption is that the corpus is mostly static and provided to the system "ahead of time" (see Section 2.1). Thus, it is feasible to consider techniques that preprocess the corpus in some manner prior to accepting user queries in an online setting. A common design for real-world systems is nightly updates to the corpus (e.g., addition, modification, or removal of texts), where it would then be possible to apply corpus-level analysis, rebuild the inverted index in a multi-stage ranking architecture, refine the ranking models, and then deploy the updated indexes and rankers to production. See Leibert et al. [2011] for an example of how one might design production infrastructure along these lines. These updates, of course, all depend on the availability of computation resources and distributed processing infrastructure, but we note that some types of analyses are embarrassingly parallel and relatively easy to scale out.
This section presents three document preprocessing techniques in this context: doc2query [Nogueira et al., 2019b] , DeepCT [Dai and Callan, 2019a] , and HDCT . All three methods share two key features: First, they push expensive neural network inference from query time to indexing time in a corpus preprocessing step. Second, they provide drop-in replacements for inverted indexes used by standard exact match scoring functions such as BM25. In scenarios where the effectiveness of these techniques alone are "good enough", queries can be executed on CPUs with low latency and high throughput; neural network inferences, typically requiring GPUs, can be completely eliminated. In scenarios where indexes produced by these techniques are used for first-stage retrieval in a multi-stage ranking architecture, they provide downstream rankers with a richer set of candidate texts to analyze.
Following discussions of document preprocessing, we introduce a technique known as target corpus pretraining, which shares in the same general idea of learning document representations that are potentially useful for text ranking.
We first begin with a general discussion of document expansion techniques, which provide the background necessary to understand doc2query, DeepCT, and HDCT. The vocabulary mismatch problem [Furnas et al., 1987] , where searchers and authors use different words to describe the same concepts, was introduced in Section 1.2.2 as a core problem in information retrieval. Any ranking technique that depends on exact matches between queries and texts suffers from this problem, and researchers have been exploring approaches to overcome the limitations of exact matching for decades. Text ranking models based on neural networks, by virtue of using continuous vector representations, offer a potential solution to the vocabulary mismatch problem because they are able to learn "soft" or semantic matches-this was already clearly demonstrated by pre-BERT neural ranking models (see Section 1.2.4).
However, in the architectures discussed so far-either the simple retrieve-and-rerank approach or multi-stage ranking-the initial candidate generation stage forms a critical bottleneck since it still depends on exact matching, for example, using BM25. A relevant text that has no overlap with query terms will not be retrieved, and hence will never be encountered by any of the downstream rankers.
In the best case, rerankers can only bring candidate texts that are deep in the ranked list up to the top of the ranked list (and are quite good at that). They, of course, cannot conjure relevant results out of thin air if none exist in the pool of candidates to begin with! In practice, it is not likely that a relevant text has no overlap with the query, 89 but it is common for relevant documents to be missing a key term from the query (for example, the document might use a synonym). Thus, the vocabulary mismatch problem can be alleviated in a brute force manner by simply increasing the depth of the candidates that are generated in first-stage retrieval. Relevant texts will show up, just deeper in the ranked list. We see this clearly in Figure 7 , where monoBERT is applied to increasing candidate sizes from bag-of-words queries scored with BM25: effectiveness increases as more candidates are examined. Nevertheless, this is a rather poor solution. The most obvious issue is that reranking latency increases linearly with the size of the candidates list under consideration, since inference needs to be applied to every candidate, although this can be mitigated by multi-stage rerankers that prune the candidates successively, as discussed in Section 3.4.
Document expansion techniques provide a potential solution to this problem. 90 The basic idea is to augment (i.e., expand) texts in the corpus with additional terms that are representative of their contents or with query terms for which those documents might be relevant. As a simple example, a text discussing automobile sales might be expanded with the term "cars" to better match the query "car sales per year in the US". In the simplest approach, these expansion terms can be appended to the end of the document, prior to indexing, and retrieval can proceed exactly as before, but on the augmented index.
A similar effect can be accomplished with query expansion, e.g., augmenting the query "car sales per year in the US" with the term "automobile". However, there are two main advantages to document expansion over query expansion:
• Documents are typically much longer than queries, and thus provides more context for a model to choose appropriate expansion terms. As we have seen from the work of Dai and Callan [2019b] (see Section 3.3.2), BERT benefits from richer contexts and is able to exploit semantic and other linguistic relations present in a piece of text.
• Most document expansion techniques are embarrassingly parallel, i.e., they are applied independently to each document. Furthermore, since document expansion can be considered a preprocessing step, the computations can be distributed over arbitrarily large clusters. This means that computationally expensive models with long inference latencies may still be practical for performing the expansion, if provided with sufficient resources. In contrast, query expansion techniques have much more restrictive latency budgets because any model inference must be applied online, for every incoming query.
There are, however, also two major disadvantages:
• Experimental cycles for exploring document expansion techniques are much longer, since each new model (or even model variant) must be applied to the entire collection, the results of which must be reindexed, before evaluations can be conducted. This means that even simple investigations such as parameter tuning can take a long time. In contrast, query expansion techniques can be explored rapidly because they do not require costly processing on the entire collection.
• Document expansion techniques increase the length of each document-since the expansion terms are (usually) appended to the documents prior to indexing. Longer documents increase query latencies, which means that candidate generation in a multi-stage ranking architecture takes longer. However, in practice, the increases in latency are usually modest; note that query expansion techniques also increase query latencies, but due to increased query lengths. 89 Although it is certainly possible for texts that contain zero query terms to be relevant to an information need, there is a closely-related methodological issue of whether test collections contain such judgments. With the pooling methodology that underlies the construction of most modern test collections (see Section 2.4), only the results of participating teams are assessed. Thus, if participating systems used techniques reliant on exact term matching, it is unlikely that a relevant document with no term overlap will ever be assessed to begin with. For this reason, high-quality test collections require diverse run submissions. 90 In this section, we intentionally switch from our preferred terminology of referring to "texts" (see Section 2.7) back to "documents", as "document expansion" is commonly used and the alternative "text expansion" can be a source of confusion (since queries are texts are well). Document expansion has a history that dates back to at least the 1980s (see overview in Section 1.2.2), with precursors arguably dating back to the late 1960s. Thus, the approach is by no means newhowever, the use of neural networks, particularly transformer architectures, has been game-changing.
The first successful application of neural networks to document expansion was introduced by Nogueira et al. [2019b] , who called their technique doc2query. The basic idea is to train a sequence-to-sequence model that, given a text from a corpus, produces queries for which that document might be relevant. This can be thought of as predictively annotating a piece of text with relevant queries. Given a dataset of (query, relevant text) pairs, which are just standard relevance judgments, a sequence-to-sequence model can be trained to generate a query given a piece of text from the corpus as input.
The predicted queries are then appended to the original texts from the corpus without any special markup to distinguish the original text from the predicted queries, forming the "expanded document". This expansion procedure is performed on every text in the corpus, and the results are indexed as usual. This index can then provide a drop-in replacement for use in first-stage retrieval in a multi-stage ranking pipeline, compatible with any of the ranking models described in this survey.
It should be no surprise that the MS MARCO passage retrieval test collection can be used as training data (thus, the method is targeted to making query predictions on passage-length texts). In terms of modeling choices, it should also be no surprise that Nogueira et al. [2019b] exploited transformers for this task. Specifically, they examined two different models:
• doc2query-base: the original proposal of Nogueira et al. [2019b] used a transformer model trained from scratch (i.e., not pretrained).
• doc2query-T5: in a follow up, Nogueira and Lin [2019] replaced the "base" non-pretrained transformer with T5 [Raffel et al., 2020] , a pretrained transformer model.
To train both models (more accurately, to fine tune, in the case of T5), the following loss is used:
where a query q consists of tokens q 0 , ..., q M , and P (y i |x) is the probability assigned by the model at the i-th decoding step to token y given the input x. Note that at training time the correct tokens q <i are always provided as input in the i-th decoding step. That is, even though the model might have predicted another token at the (i − 1)-th step, the correct token q i−1 will be fed as input to the current decoding step. This training scheme is called teacher forcing or maximum likelihood learning and is commonly used in text generation tasks such as machine translation and summarization.
At inference time, given a piece of text as input, multiple queries can be sampled from the model using top-k random sampling [Fan et al., 2018a] . In this sampling decoding method, at each decoding step a token is sampled from the top-k tokens with highest probability from the model. The decoding stops when an special "end-of-sequence" token is sampled. In contrast to other decoding methods such as greedy and beam search, top-k sampling tends to generate more diverse texts, with diversity increasing with greater values of k [Holtzman et al., 2019] . Note that the k parameter is independent of the number of sampled queries; for example, we can set k = 10 and sample 40 queries from the model. In other words, each inference pass with the model generates one predicted query, and typically, each text in the corpus is expanded with many queries.
Results on the MS MARCO passage retrieval test collection are shown in The effectiveness differences between doc2query with the "base" (non-pretrained) transformer and the (pretrained) T5 model with BM25 retrieval are clearly seen in row (1c) vs. row (1b). Note that both models are trained using exactly the same dataset. It should come as no surprise that T5 is able to make better query predictions. While the T5 condition used a larger model that has more parameters than the base transformer, overparameterization of the base transformer can lead to poor predictions, and it appears clear that pretraining makes the crucial difference, not model size per se. With BM25 + RM3, row (2c) vs. row (2b), the gap between doc2query-T5 and doc2query-based is reduced, but these experiments exhibit the same issue as with the monoBERT experiments (see Section 3.2), where sparse judgments are not able to properly evaluate the benefits of query expansion (more below). Table 15 provides two additional points of reference: monoBERT, shown in row (4) as well as the best contemporaneous non-BERT model, shown in row (3). The effectiveness of doc2query is far below that of monoBERT, but is about 50× faster, since the technique is still using keyword search with BM25. The modest increase in query latency is due to the fact that the expanded texts are longer. The comparison to row (3) shows that doc2query is able to approach the effectiveness of non-BERT neural models (at the time the work was published) solely with document expansion, without any neural network inference at query time. Experiments also show that doc2query improves Recall@1k, which means that more relevant texts will be available to downstream rerankers when used in a multi-stage ranking architecture, thus potentially improving end-to-end effectiveness.
Evaluation results of doc2query on the TREC 2019 Deep Learning Track passage retrieval test collection are shown in Table 16 ; these results have not been reported elsewhere. The primary goal of this experiment is to quantify the effectiveness of doc2query using non-sparse judgments, similar to the experiments reported in Section 3.2. As discussed previously, sparse judgments from the MS MARCO passage retrieval test collection are not sufficient to capture improvements attributable to RM3, whereas with the TREC 2019 Deep Learning Track passage retrieval test collection, it becomes evident that query expansion improves effectiveness over bag-of-words queries. Similarly, results in Table 16 show that on an index that has been augmented with doc2query predictions, BM25 + RM3 is more effective than just BM25 alone; compare row (2b) vs. row (1b) and row (2c) vs. row (1c). Based on these judgments, doc2query-T5 with BM25 + RM3 achieves the highest effectiveness.
Input: July is the hottest month in Washington DC with an average temperature of 27 • C (80 • F) and the coldest is January at 4 • C (38 • F) with the most daily sunshine hours at 9 in July. The wettest month is May with an average of 100mm of rain.
Target query: what is the temperature in washington This is consistent with the literature, and these results show that doc2query gains are additive with improvements from pseudo-relevance feedback. Table 16 shows two additional comparison conditions: row (3), which applies monoBERT to rerank BM25 + RM3 results (also from Anserini), and row (4), the top-scoring submission to TREC 2019 Deep Learning Track passage retrieval task [Yan et al., 2019] . While doc2query does not achieve anywhere near the level of effectiveness as reranking with monoBERT, row (2c) vs. row (3), this is entirely expected, and the much faster query latency of doc2query has already been pointed out. We further note that the work of Yan et al.
[2019] adopted a variant of doc2query (and further exploits ensembles), which provides independent evidence supporting the effectiveness of document expansion via query prediction.
Where exactly are the gains of doc2query coming from? Figure 14 provides three examples from the MS MARCO passage corpus, showing query predictions by both the base transformer as well as T5. The predicted queries seem quite reasonable based on manual inspection. Interestingly, both models tend to copy some words from the input text (e.g., "washington dc" and "river"), meaning that the models are effectively performing term reweighting (i.e., increasing the importance of key terms). Nevertheless, the models also produce words not present in the input text (e.g., weather), which can be characterized as expansion by adding synonyms and related terms.
To more accurately quantify these effects, it is possible to measure the proportion of terms predicted by doc2query-T5 that already exist in the original text (i.e., are copied) vs. terms that do not exist in the original text (i.e., are new terms). Here, we describe the results of exactly such an analysis, which has not been previously published. Excluding stopwords, which corresponds to 51% of the predicted query terms, we find that 31% are new while the rest (69%) are copied. Table 17 presents the results of an ablation analysis: starting with the original text, we add only the new terms, row (2a); only the copied terms, row (2b); and both, row (2c). Each variant of the expanded corpus was then indexed as before, and results of bag-of-words keyword search with BM25 (using Anserini) are reported. The final condition is exactly the same as row (1c) in Table 15 , repeated here for convenience.
We see that expansion with only new terms yields a small improvement over just the original texts. Expanding with copied terms alone provides a greater gain, indicating that the effects of term reweighting appear to be more impactful than attempts to enrich the vocabulary. However, combining both types of terms yields a big jump in terms of effectiveness, showing that the evidence is complementary. Interestingly, the gain from both types of terms together is greater than the sum of the gains from each individual contribution in isolation. This can be characterized with the popular adage, "the whole is greater than the sum if its parts", and suggests complex interactions between the two types of terms that we do not fully understand. Usually, the gains from the combination of two innovations is smaller than the sum of the gain for each applied independently; see Armstrong et al. [2009] for more discussion. Finally, row (3) in Table 17 presents the results of an interesting condition: what if we discarded the original texts and just indexed only the expansions (i.e., the predicted queries). We see that effectiveness is surprisingly high, only slightly worse than the full expansion condition. In other words, it seems like the original texts can, to a large extent, be replaced by the predicted queries from the perspective of bag-of-words search. In effect, we have converted a traditional search task into a community question answering task! Takeaway Lessons. To sum up, document expansion with doc2query augments texts with potential queries, thereby mitigating vocabulary mismatch and reweighting existing terms based on predicted importance. The expanded collection can be indexed and used exactly as before-either by itself or as part of a multi-stage ranking architecture. Perhaps due to its simplicity and effectiveness, doc2query has been adapted and successfully applied to other tasks, including scientific document retrieval [Boudin et al., 2020] , creating artificial in-domain retrieval data [Ma et al., 2020] , and helping users in finding answers in product reviews [Yu et al., 2020b] .
Document expansion with doc2query, in effect, shifts computationally expensive inference with neural networks from query time to indexing time. As a drop-in replacement for the original corpus, keyword search latency increases only modestly due to the increased length of the indexed texts. The downside of doc2query is much more computationally intensive data preparation prior to indexing: for each text in a corpus, multiple inference passes are needed to generate the expanded queries. If the corpus is large (e.g., billions of documents), this method can still become prohibitively expensive. 91 For researchers who work with standard test collections, however, this usually isn't an issue because Nogueira and Lin [2019] have made their query predictions on standard corpora publicly available for download, making doc2query pretty close to a "free boost" that can be applied to a range of other techniques.
Results from doc2query show that document expansion has two distinct but complementary effects: novel expansion terms that are not present in the original text and copies of terms that are already present in the text. The duplicates have the effect of reweighting terms in the original text, but using a sequence-to-sequence model to generate terms seems like an inefficient and roundabout way of achieving this effect.
What if we are able to directly estimate the importance of a term in the context that the term appears in? This is the premise of the Deep Contextualized Term Weighting (DeepCT) framework [Dai and Callan, 2019a] . Consider BM25 scores (Section 1.2.2), which at a high level comprises a term frequency and a document frequency component. Setting aside length normalization, 92 the term frequency (i.e., the number of times the term appears in a particular text) is the primary feature that attempts to capture the term's importance in the text, since the document frequency component of 
Best Answer: a troll is generally someone who tries to get attention by posting things everyone will disagree, like going to a susan boyle fan page and writing susan boyle is ugly on the wall. they are usually 14-16 year olds who crave attention.
Query what values do zoos serve
Zoos serve several purposes depending on who you ask. 1) Park/Garden: Some zoos are similar to a botanical garden or city park. They give people living in crowded, noisy cities a place to walk through a beautiful, well maintained outdoor area. The animal exhibits create interesting scenery and make for a fun excursion.
There 
Genomics in Theory and Practice. What is Genomics. Genomics is a study of the genomes of organisms. It main task is to determine the entire sequence of DNA or the composition of the atoms that make up the DNA and the chemical bonds between the DNA atoms. BM25 is the same for that term across different texts (with the same length). Terms can have the same term frequency in different texts but differ in the "importance" they play.
A number of motivating examples taken from Dai and Callan [2019a] are presented in Figure 15 . In the first example, the "off-topic" (non-relevant) passage actually has more occurrences of the key query terms "susan" and "boyle", yet it is clear that the first passage provides a better answer. The second and third examples similarly reinforce the observation that term frequencies alone are often insufficient to separate relevant from non-relevant passages. In the third example, "atoms" appear twice in both passages, but it seems clear that the first passage is relevant while the second is not.
To operationalize these intuitions, the first and most obvious question that must be addressed is: How should term importance weights or scores (we use these two terms interchangeably) be defined? Dai and Callan [2019a] proposed a simple measured called query term recall, or QTR:
where |Q d | as the set of queries that are relevant to document d, and |Q d,t | is the subset of |Q d | that contain term t. The importance score y t,d for each term t in d can then be defined as follows:
The score y t,d is in the range [0 . . . 1]. At the extremes, y t,d = 1 if t occurs in all queries for which d is relevant, and y t,d = 0 if t does not occur in any query relevant to d. Going back to the examples in Figure 15 , "susan" and "boyle" would receive lower importance weights in the second passage because it doesn't come up in queries about "susan boyle" as much as the first passage. With appropriate scaling, these weight can be converted into drop-in replacements of term frequencies, replacing the term frequency values that are stored in a standard inverted index. In turn, a DeepCT index can be used in the same way as any other standard bag-of-words inverted index, for example, to provide candidate texts for first-stage retrieval.
Having thus defined term importance weights using query term recall, it then becomes relatively straightforward to formulate the prediction of these weights as a regression problem. Not surprisingly, BERT can be exploited for this task. More formally, DeepCT uses a BERT-based model that receives as input a text d and outputs an importance score y t,d for each term t in d. The goal is to assign high scores to terms that are central to the text, and low scores to less important terms. These scores are computed by a regression layer as:ŷ
where w is a weight vector, b is a bias term, and T t,d is the contextual embedding of term t in the text.
Like doc2query, DeepCT is trained using a dataset of (query, relevant text) pairs (the MS MARCO passage retrieval test collection). The BERT model and the regression layer are trained end-to-end to minimize the following mean squared error (MSE) loss:
whereŷ t,d and y t,d have already been defined. Note that the BERT tokenizer often splits words in the texts into subwords (e.g., "telescope" might be tokenized into "tele" and "##scope"). DeepCT uses the weight for the first subword as the weight of the entire word; other subwords are ignored when computing the MSE loss.
Once a regression model is trained, inference is applied to computeŷ t,d for each text d in the corpus. These weights are then rescaled from [0.
.1] to integers between 0 and 100, so they resemble term frequencies in standard bag-of-words retrieval methods. Finally, the texts are indexed using these rescaled term weights using a simple trick that does not require changing the underlying indexing algorithm to support custom term weights. New "pseudo-documents" are created in which terms are repeated the same number of times as their importance weights. For example, if the term "boyle" is assigned a weight of four, it is repeated four times, becoming "boyle boyle boyle boyle" in this new pseudo-document. A new corpus comprising these pseudo-documents, in which the repeated terms are all concatenated together, is then indexed like any other corpus. Retrieval is performed on this index as with any other bag-of-words query, 93 although it is important to retune parameters in the scoring functions.
Experiment results for DeepCT using BERT Base for regression on the MS MARCO passage retrieval test collection are presented in Table 18 , copied from Dai and Callan [2019a] . The obvious point of comparison is doc2query, and thus we have copied figures from Table 15 and Table 17 . Note that doc2query-base, row (1b), predates DeepCT, and is included in the authors' comparison, but doc2query-T5 was developed after DeepCT.
How do the two approaches compare? It appears that DeepCT is more effective than the "base" version of doc2query (i.e., training a sequence-to-sequence transformer from scratch) but is not as effective as doc2query based on T5, which benefits from pretraining. Evaluation in terms of Recall@1k tells a consistent story: all three techniques increase the number of relevant documents that are available to downstream rerankers, and the effectiveness of DeepCT lies between doc2querybase and doc2query-T5. In row (1d), we repeat the results of the doc2query-T5 ablation analysis in Table 17 where only repeated expansion terms are included. This discards the effects of new terms, bringing the comparison into closer alignment with DeepCT. Comparing row (2) with row (1d), we see that DeepTC's principled approach to reweighting terms is more effective than relying on T5 to reweight terms indirectly via copied terms in independent query predictions.
It is worth noting that a comparison between the two methods is not entirely fair since doc2query's T5-base model is twice the size of DeepCT's BERT Base model, and it was pretrained on a larger corpus. Thus, we cannot easily separate the impact on effectiveness of simply having a bigger model, as opposed to fundamental characteristics of the underlying techniques.
While not as effective as the best variant of doc2query, DeepCT does have a number of advantages: its model is more lightweight in terms of neural network inference and thus preprocessing the entire corpus with DeepCT (which is necessary prior to indexing) is much faster. DeepCT uses an encoderonly model (e.g., BERT), which tends to be faster than encoder-decoder models used by doc2query since there is the additional output sequence generation phase. Furthermore, DeepCT requires only one inference pass per text to compute term importance weights for all terms in the text, whereas doc2query requires an inference pass to generate each query prediction, which must be repeated multiple times (typically 5 to 40). 94
The other major difference between DeepCT and doc2query is that DeepCT is restricted to reweighting terms that are already present in a text, whereas doc2query can augment the existing text with new terms, thus potentially helping to bridge the vocabulary mismatch gap. The higher recall observed with doc2query-T5 in Table 18 is perhaps attributable to these expansion terms. The addition of new terms not present in the original texts, however, increases keyword search latency by a modest amount due to the increased length of the indexed texts. In contrast, the performance impact of DeepCT is negligible, as experimentally validated by . 95
Takeaway Lessons. At a high level, doc2query and DeepCT represent two different realizations of the insight that transformers can be applied to preprocess a corpus in a manner that improves retrieval effectiveness. Both techniques share two key features: they eliminate the need for expensive neural network inference at query time (as inference is pushed into the preprocessing stage), and they provide drop-in replacements for keyword search. For certain applications, we might imagine that bag-of-word queries over doc2query or DeepCT indexes might be "good enough", and results can be directly returned to users (without additional reranking). In this case, we have completely eliminated query-time dependencies on inferences using neural networks (and their associated hardware requirements). Alternatively, either doc2query or DeepCT can be used for first-stage retrieval in a multi-stage reranking pipeline to improve recall, thus providing downstream rankers with more relevant documents to work process and potentially improving end-to-end effectiveness.
In follow-up work building on DeepCT, proposed HDCT, a context-aware hierarchical document term weighting framework. Similar to DeepCT, the goal is to estimate a term's context-specific term importance based on contextual embeddings from BERT, which is able to capture complex syntactic and semantic relations within local contexts. Like DeepCT, these term importance weights (or scores) are mapped into integers so that they can be directly interpreted as term frequencies, replacing term frequencies in a standard bag-of-words inverted index.
Like much of the discussion in Section 3.3, HDCT was designed to address the length limitations of BERT. DeepCT did not encounter this issue because it was only applied to paragraph-length texts such as those in the MS MARCO passage corpus. As we've already discussed extensively, BERT has trouble with input sequences longer than 512 tokens for a number of reasons. The obvious solution, of course, is to split texts into passages.
To process long texts, HDCT splits them into passages comprising consecutive sentences that are up to about 300 words. After processing each passage with BERT, the contextual embedding of each 94 Although this is easily parallelizable on a cluster. 95 Note that it is not a forgone conclusion that term reweighting will retain the same performance profile in bag-of-word querying (i.e., query latencies and their distributions) compared to "normal" term frequencies.
While the terms have not changed, the term weights have, which could affect early-exit and other optimizations in modern query evaluation algorithms (which critically depend on the relative weights between terms in the same text). Thus, the performance impact of term weighting requires empirical examination and cannot be derived from first principles; see for an in-depth and nuanced look at these effects.
term is fed into a linear layer to map the vector representation into a scalar weight:
where T BERT (t, p) is the contextual embedding produced by BERT for term t in passage p, w is the weight vector, and b is the bias. Like DeepCT, predicting the importance weight of term t in passage p, denotedŷ t,p , is formulated as a regression problem. 96 By construction, ground truth labels are in the range [0, 1] (see below), and thus so are the predictions, y t,p ∈ [0, 1]. They are then scaled into an integer as follows:
where N = 100 retains two digit precision and taking the square root has a smoothing effect. 97 The weight tf BERT (t, p) captures the importance of term t in passage p according to the BERT regression model, rescaled to a term frequency-like value.
There are still a few more steps before we arrive at document-level tf BERT weights. Thus far, we have a bag-of-words vector representation of each passage p:
Gathering all the results from each passage yields a sequence of bag-of-words passage vectors:
{P-BoW HDCT (p 1 ), P-BoW HDCT (p 2 ), . . . , P-BoW HDCT (p m )}.
Finally, the importance weight for each term t in document d is computed as:
where pw i is weight for the passage p i . experimented with two ways for computing the weight of the passage: in the "sum" approach, pw i = 1, and in the "decay" approach, pw i = 1/i. The first approach considers all passages equal, while the second discounts passages based on their position, i.e., passages near the beginning of the text are assigned a higher weight. Although "decay" is slightly more effective on newswire documents than "sum", the authors concluded that "sum" appears to be more robust, and also works well with web pages. At the end of all these processing steps, each (potentially long) text is converted into a bag of terms, where each term is associated with an integer importance weight.
Given this setup, the only remaining issue is the "ground truth" y t,p labels for the term importance weights. Recall that in DeepCT, these scores are derived from query term recall based on (query, relevant text) pairs, from the MS MARCO passage retrieval test collection. There are two issues for this approach:
1. Labeled datasets at this scale are costly to build.
2. Relevance judgments are made at the document level, but the HDCT regression problem is formulated at the passage level; see Eq. (38).
Thus, explored weak supervision techniques to automatically generate training labels. Note that the second motivation is exactly the same issue Akkalyoncu dealt with in Birch, and the findings here are consistent (see Section 3.3.1). In the end, experiments with HDCT found that automatically deriving global (document-level) labels appears to be sufficient for training local (passage-level) term importance predictors; BERT's contextual embeddings appear to generate high-quality local weights at the passage level. This is similar to the "don't worry about it" approach adopted by BERT-MaxP (see Section 3.3.2).
Two techniques were proposed for generating term importance weights for training: 96 Note that although DeepCT and HDCT are by the same authors, their two papers use slightly different notation, in some cases, for the same ideas; for example Eq. (38) and Eq. (36) both express term importance prediction as regression. Nevertheless, for clarity, we preserve the notation used in each of the original papers. 97 Note that DeepCT is missing this square root. [Jin et al., 2002] .
Having defined the target labels y t,p , the BERT regression model can be trained. As with DeepCT, HDCT is trained end-to-end to minimize the following mean squared error (MSE) loss.
An evaluation of HDCT on the development set of the MS MARCO document retrieval test collection is shown in Table 19 , copied from . Their paper presented evaluation on web collections as well as a number of detailed analyses and ablation studies, but for brevity here we only convey the highlights. These results used BERT Base . Statistically significant differences are denoted by the superscripts, e.g., row (2a) is significantly better than row (1) and row (2b).
As the baseline, built an ensemble of BM25 rankers on different document fields: title, body, and URL in the case of MS MARCO documents. This is shown as row (1). The effectiveness of the HDCT passage regression model for predicting term importance, trained on the MS MARCO document retrieval test collection, which contains approximately 370k (query, relevant document) pairs, is shown in row (3). This condition provides an upper bound for the weak supervision techniques, since the labels are provided by humans. Row (2a) shows the effectiveness of using document titles for weak supervision. Rows (2b) and (2c) show the effectiveness of using pseudo-relevant documents, with different queries. In (2b), the AOL query log [Pass et al., 2006] is used, while in (2c), queries from the training set of the MS MARCO document retrieval test collection were used (but without the corresponding labels). The first might be characterized as an "out of domain" query log, as those queries differ from the MS MARCO queries; the second can be characterized as weak supervision with the same types of queries as those used in the evaluation. We see that weak supervision with MS MARCO queries is more effective than using document metadata, but using the AOL query log is worse than simply using document metadata.
Takeaway Lessons. Building on DeepCT, HDCT provides two additional important lessons. First, the approach provides relatively easy techniques to address the length limitations of BERT, thus allowing the same ideas behind DeepCT to be applied to longer texts. Second, in the absence of large labeled datasets for learning term importance weights, weak supervision can be helpful. The conditions in row group (2) in Table 19 shows that weak supervision with titles can be effective; these represent statistically significant gains that come "for free". Row (2c) shows that having an "in domain" sample of queries (even without relevance judgments) is greatly beneficial, and that an "out of domain" sample of queries (AOL queries) can be worse than just using titles. Of course, manual relevant judgments can lead to large and significant gains in effectiveness, but weak supervision with labels from pseudo-relevant document gets us around 65% of the gains from fully-supervised approach. 
The document preprocessing techniques described above share in the goal of manipulating document representations to address the vocabulary mismatch problem, thereby increasing retrieval effectiveness. However, these techniques primarily tackle limitations in the candidate generation stage of a multistage ranking pipeline, with the hope that by increasing recall in first-stage retrieval, downstream rerankers will have more relevant candidates to work with. In most of the techniques we have presented so far, training a reranker (say, monoBERT) more accurately refers to fine-tuning an already pretrained model (see Section 3.2). That is, we begin with a pretrained model and directly fine tune on labeled data drawn from the same distribution as our final ranking task (i.e., in-domain labeled data). We discuss two general techniques that can be introduced before this fine tuning begins to improve end-to-end effectiveness:
• Additional pretraining on the target corpus.
• Fine-tuning on out-of-domain relevance judgments.
It is important to note that the first technique takes advantage of self supervision, while the second depends on labeled data have been created for other purposes. In keeping with the theme of this section, these two techniques can be characterized as efforts to better align representations (in this case, of the models) with the ranking task at hand.
The "base" transformers models such as the BERT Base and BERT Large checkpoints directly downloadable from Google are typically pretrained (e.g., based on the mask language model objective) on general domain corpora: for example, BERT uses the BooksCorpus [Zhu et al., 2015] and well as Wikipedia. While there may be some overlap between these corpora and the target corpus over which ranking is performed, they may nevertheless differ in terms of vocabulary composition, syntactic structure, style, and numerous other factors. This is particularly the case for ranking in specialized domains, which we separately discuss in Section 3.7. Thus, it may be helpful to perform additional pretraining on the target corpus. More concretely, the model should be provided the chance to learn more about the distribution of the terms and their co-occurrences prior to learning how to rerank them.
Put differently, the reranker should be given an opportunity to "see" all the texts in a corpus before starting to compute relevance signals. It is important here to emphasize that pretraining requires only access to corpus we are searching, and does not require any queries or relevance judgments. Nogueira et al. [2019a] did exactly this, which they called target corpus pretraining (TCP), in the context of their multi-stage architecture, as discussed in Section 3.4.1. Instead of using Google's BERT checkpoints as the starting point of fine tuning, they began by additional pretraining on the MS MARCO passage corpus using the same objectives from the original BERT paper, i.e., masked language modeling and next sentence prediction. These results are shown in Table 20 : row (2b) and row (3b) show that TCP is able to improve over ablated conditions without TCP, row (2a) and row (3a), respectively, but the improvement is modest at best (less than one point). Nevertheless, these improvements "come for free' in the sense of not requiring any labeled data, and so adopting this technique might be worthwhile in certain scenarios. These results are in line with recent work that shows improvements with target corpus pretraining over out-of-domain corpus pretraining for a variety of language analysis tasks [Beltagy et al., 2019 , Raffel et al., 2020 , Gururangan et al., 2020 .
In addition to target corpus pretraining, researchers have also explored "stage-wise" or "multi-phase" fine tuning for text ranking applications. The idea is to leverage out-of-task or out-of-domain relevance judgments to fine-tune a model before fine-tuning on labeled data drawn from the same distribution as the final task (i.e., in-domain labeled data). In the same way that target corpus pretraining gives the model a sense of what the texts "look like" before attempting to learn relevance signals, this fine-tuning technique attempts to provide the model "general" notions of relevance matching before using task specific data.
In fact, we have already seen this technique working in Birch, discussed in Section 3.3.1. Recall that Birch's solution to BERT's length limitations was to train the model with out-of-domain data, and deploy the ranker in a zero shot manner. The model actually skips fine tuning on in-domain data, but the idea is the same. This technique of iteratively fine-tuning on multiple supervised datasets in sequence has been explored by many researchers: Garg et al. [2020] call this the "transfer and adapt" (TANDA) approach. Dai and Callan [2019a] first fine-tune on data from search engine logs before further fine-tuning on TREC collections. Applied to question answering Yang et al. [2019d] called this "stage-wise" fine-tuning; cf. . Similarly, PARADE first fine-tunes on the MS MARCO passage retrieval test collection before fine-tuning further on collection-specific relevance judgments . Nor is this technique limited to text ranking problems: to our knowledge, the first reported instance of sequential fine-tuning with multiple labeled datasets is by Phang et al. [2018] , on a range of natural language inference tasks. This work predates all the above cited papers focused specifically on text ranking.
We have already seen instances of BERT's ability to handle zero-shot text ranking and domain transfer of relevance matching models. These two findings become particularly important when transformers are applied to domain-specific applications, such as legal texts and scientific articles. We defer detailed discussions of these applications until Section 3.7.
Takeaway Lessons. Beyond the standard paradigm of fine-tuning pretrained transformer models with task-specific labeled data, there are two additional tricks that can boost effectiveness. Prior to any fine-tuning, we can perform additional (self-supervised) pretraining on the target corpus. Prior to fine-tuning on in-domain labeled data, we can first fine-tune with out-of-domain labeled data. While both tricks are beneficial, details of how to properly sequence their application (e.g., how many epochs to run, how many and what order to apply out-of-domain datasets) to maximize the benefits of additional pretraining and transfer learning effects remain somewhat of an art, and the process typically involves lots of trial and error. While we understand at a high level why these techniques work, more research is required to sharpen our understanding so that expected gains can be accurately predicted and modeled without the need to conduct extensive experiments.
All of the ranking models discussed so far in this survey are still primarily built around BERT, even if they incorporate other architectural components, such as interaction matrices in CEDR (Section 3.3.3) or another stack of transformers in PARADE (Section 3.3.4). There are, however, many attempts to move beyond BERT to explore other transformer models, which is the focus of this section.
At a high level, efforts to improve ranking models can be characterized as attempts to make ranking better, attempts to make ranking faster, attempts to accomplish both, or attempts to find other operating points in the effectiveness/efficiency tradeoff space. Improved ranking effectiveness is, of course, a perpetual quest and needs no elaboration. Attempts to make text ranking models faster can be motivated by, among many sources, Figure 16 , taken from Hofstätter and Hanbury [2019], which shows not only the effectiveness of different neural ranking models on the MS MARCO passage retrieval task but query latency (millisecond per query) as well. Note the gap in the x axis! Pre-BERT models can be deployed for real-world applications with minimal modifications, but it is clear that naïve production deployments of BERT are impractical or hugely expensive in terms of required hardware resources. In other words, BERT is good but slow: can we trade off a bit of quality for better performance?
The presentation of this section is organized roughly in increasing "distance from BERT." Admittedly, what's BERT and what's "beyond BERT" is somewhat an arbitrary distinction, and in fact this section naturally flows into the discussions in Section 4 on continuous dense representations for ranking, which are motivated by many of the issues discussed here. These classifications represent primarily our judgment for expository purposes since we are constrained by the linear nature of prose, and shouldn't be taken as any sort of definitive categorization. We begin by discussing simple modifications to existing models that "swap out" BERT for one of its variants and efforts to distill BERT into smaller models (Section 3.6.1). An attempt to design transformer-based architectures specifically for text ranking from the ground up-the Transformer Kernel (TK) and Conformer Kernel (CK) models-are discussed next (Section 3.6.2). Finally, we turn our attention to ranking with pretrained sequence-to-sequence models using T5 (Section 3.6.3).
Recognizing that the distinction between a simple architectural variant of BERT and an entirely new model can be somewhat fuzzy, in this section we discuss two common strategies for improving either the effectiveness or efficiency of BERT as applied to text ranking:
1. starting with a BERT variant that has been pretrained with an improved procedure (e.g., data size, hyperparameters), primarily to improve effectiveness; and, 2. distilling BERT into a smaller model, primarily to achieve a better tradeoff between effectiveness and efficiency.
Examples of models that can be characterized as "BERT with improved pretraining" include RoBERTa, ALBERT, and ELECTRA. RoBERTa [Liu et al., 2019c] represents a replication study of BERT's pretraining procedures, with additional explorations of many design choices made in the original paper. In particular, the authors argued that Google's original BERT model was significantly under-trained; by modifying several hyperparameters and by removing the next sentence prediction task, RoBERTa is able to match or exceed the effectiveness of BERT on a variety of natural language analysis tasks. ALBERT [Lan et al., 2020] reduces the memory footprint of BERT by tying the weights in its transformer layers together (i.e., it uses the same weights n times rather than separate weights for each layer). ELECTRA [Clark et al., 2020] attempts to improve BERT pretraining by substituting its masked language model pretraining task with a replaced token detection task, in which the model predicts whether a given token has been replaced with a token produced by a separate generator model. As a result, the contextual representations learned by ELECTRA outperform the ones learned by BERT on various language analysis tasks given the same model size, data, and compute. While none of these variants have yet reached the popularity of BERT, experimental results (particularly with RoBERTa and ELECTRA) suggest that improving pretraining is a promising research direction.
For the most part, these BERT variants have been evaluated on the same language analysis tasks as BERT (see Section 3.1), although applications to text ranking are straightforward and researchers have indeed begun to pursue this avenue. results have not been previously published, but the experimental setup is exactly the same as in Section 3.2 and the monoBERT Large results are copied from row (3b) in Table 5 . We see that although Liu et al. [2019c] found RoBERTa to achieve higher effectiveness across a range of NLP tasks, these improvements do not appear to carry over to text ranking, as monoRoBERTa Large reports a slightly lower MRR@10. This finding suggests that information access tasks need to be examined independently from the typical suite of language analysis tasks employed by NLP researchers.
In the context of PARADE, investigated replacing BERT Base with ELECTRA Base . These results (copied from the paper) are shown in Table 21 , where ELECTRA Base yields a small increase in effectiveness for both title and description queries on Robust04 and Gov2 test collections; the BERT Base results are copied from row (5d) in Table 12 . These improvements come (nearly) "for free" since they require only trivial modeling changes, and demonstrate the potential of exploring BERT variants.
Shifting topics, knowledge distillation refers a general set of techniques where a smaller student model learns to mimic the behavior of a larger teacher model [Ba and Caruana, 2014, Hinton et al., 2015] . Generally, the goal is for the student model to achieve comparable effectiveness on a particular task but more efficiently (e.g., lower inference latencies, fewer model parameters, etc.). While knowledge distillation is model agnostic and researchers have explored this approach for many years, to our knowledge Tang et al. [2019] was the first to apply the idea to BERT, demonstrating knowledge transfer between BERT and much simpler models such as single-layer BiLSTMs. A much simpler RNN-based student model, of course, cannot hope to achieve the same level of effectiveness as BERT, but if the degradation is acceptable, inference can be accelerated by an order of magnitude or more. These ideas have been extended by many others [Sun et al., 2019 , Liu et al., 2019b , Sanh et al., 2020 , with a range of different student models, including smaller versions of BERT.
Recently, knowledge distillation has been applied to text ranking as researchers have investigated whether the efficiency of BERT can be improved by distilling a larger trained (BERT) model into a smaller (but still BERT-based) one [Gao et al., 2020b . To encourage the student model to mimic the behavior of the teacher model, one common distillation objective is the mean squared error between the student's and teacher's logits [Tang et al., 2019 , Tahami et al., 2020 ]. The student model can be fine-tuned with the linear combination of the student model's cross-entropy loss and the distillation objective as the overall loss:
where L CE is the cross-entropy loss, r t and r s are the logits from the teacher and student models, respectively, and α is a hyperparameter. As another approach, TinyBERT proposed a distillation objective that additionally considers the mean squared error between the two models' embedding layers, transformer hidden states, and transformer attention matrices [Jiao et al., 2019] . Gao et al. [2020b] observed that distillation can be applied to both a BERT model that has already been fine-tuned for relevance classification ("ranker distillation") and to pretrained but not yet fine-tuned BERT itself ("LM distillation"). Concretely, this yields three possibilities: 1. apply distillation so that a (randomly initialized) student model learns to directly mimic an already fine-tuned teacher model using the distillation objective above ("ranker distillation"), 2. apply LM distillation into a student model followed by fine-tuning the student model for the relevance classification task ("LM distillation + fine-tuning"), or 3. apply LM distillation followed by ranker distillation ("LM + ranker distillation").
Operationally, the third approach can be thought of as the same as the first approach, except with a better initialization of the student model. The relative effectiveness of these three approaches is an empirical question. To answer this question, Gao et al. [2020b] used the TinyBERT distillation objective to distill a BERT Base model into smaller transformers: a six-layer model with a hidden dimension of 768 or a four-layer model with a hidden dimension of 312. Both the student and teacher models are designed as relevance classifiers (i.e., monoBERT).
Track passage retrieval test collection are shown in Table 23 , with results copied Gao et al. [2020b] . The six-layer and four-layer student models are shown in row groups (2) and (3), respectively, and the monoBERT Base teacher model is shown in row (1). The (a), (b), (c) rows of groups (2) and
(3) correspond to the three approaches presented above. The final column shows inference latency measured on an NVIDIA RTX 2080Ti GPU.
We see that ranker distillation alone performs the worst; the authors reported a statistically significant decrease in effectiveness from the teacher model across all metrics and both test collections. Both LM distillation followed by fine-tuning and LM distillation followed by ranker distillation led to student models comparable to the teacher in effectiveness. We see that in terms of MRR, "LM + ranker distillation" outperforms "LM distillation + fine-tuning" on the MS MARCO passage retrieval test collection, but the other way around for the TREC 2019 Deep Learning Track document retrieval test collection; note though, that the former has far more queries than the latter. Overall, the six-layer distilled model can perform slightly better than the teacher model while being twice as fast, 98 whereas the four-layer distilled model gains a 9× speedup in exchange for a small decrease in effectiveness.
As another example of explorations in knowledge distillation, investigated how well their PARADE model performs when distilled into student models that range in size. Specifically, they examined two approaches: Table 24 : The effectiveness of training PARADE using a smaller BERT vs. distilling a BERT Base PARADE teacher into smaller BERT models on Robust04 title queries. Inference times are on a Google TPU v3-8. The symbol † indicates a significant improvement of a "Distill" model over the corresponding "Train" model, as determined by a paired t-test (p < 0.05).
same as Row (5d) in Table 12 . Rows (2-8) present the distillation results: The "Train" column shows the results of training PARADE with BERT models of different sizes. This corresponds to the LM distillation plus fine-tuning setting from Gao et al. [2020b] (except the full PARADE model involves more than just fine tuning). The "Distill" column indicates the result of distilling PARADE from a teacher using BERT Base , into smaller, already distilled student. This corresponds to the LM distillation plus ranker distillation setting from Gao et al. [2020b] . Inference times were calculated using a Google TPU v3-8 with a batch size of 32. The symbol † indicates a significant improvement of a "Distill" model over the corresponding "Train" model, as determined by a paired t-test (p < 0.05).
Comparing the "Train" and "Distill" columns, it is clear that distilling into a smaller model is preferable to training a smaller model directly. The models under the ranker distillation condition are always more effective than the models that are trained directly, and this increase is statistically significant in the majority of cases. These results are consistent with the finding of Gao et al. [2020b] , at least on the MS MARCO passage retrieval test collection.
In rows (2) and (3), we see that reducing the number of transformer encoder layers in BERT Base under the "Train" condition sacrifices only a tiny bit of nDCG@20 for noticeably faster inference. However, the "Distill" versions of these models perform comparably to the original BERT Base version, indicating that distillation into a "slightly smaller" model can improve efficiency without harming effectiveness. The same trends continue with smaller BERT variants, with effectiveness decreasing as the model size decreases. We also see that ranker distillation is consistently more effective than directly training smaller models. The difference between the teacher and ranker-distilled models becomes statistically significant from row (4). This indicates that ranker distillation can be used to eliminate about a quarter of PARADE's parameters and reduce inference latency by about a third without significantly harming the model's effectiveness.
The papers of Gao et al. [2020b] and , unfortunately, explore different datasets and different metrics with no overlap-thus preventing a direct comparison. Furthermore, there are technical differences in their approaches: Gao et al. [2020b] began with TinyBERT's distillation objective [Jiao et al., 2019] to produce their smaller BERT models. On the other hand, Li et al.
[2020a] used as starting points the pre-distilled models provided by Turc et al. [2019] . Since the starting points differ, it is not possible to separate the impact of the inherent quality of the smaller BERT models from the impact of the PARADE aggregation mechanisms in potentially compensating for a smaller and less effective BERT model. Nevertheless, both papers seem to suggest that finetuning a smaller model directly is less effective than distilling into a smaller model from a fine-tuned (larger) teacher, although the evidence is equivocal from Gao et al. [2020b] because only one of the two test collections support this observation. However, beyond text ranking, we find broader complementary support: results on NLP tasks show that training a larger model and then compressing it is more computationally efficient than spending the comparable resources directly training a smaller model [Li et al., 2020b] . We also note the connection here with the so-called "Lottery Ticket Hypothesis" Frankle and Carbin [2019], Yu et al. [2020a] , although more research is needed here to fully reconcile all these related threads of work.
Takeaway Lessons. We draw two high-level lessons from this discussion: First, text ranking applications can potentially benefit from advances in BERT model variants, which are mostly evaluated on traditional natural language analysis tasks such as paraphrase detection and named-entity recognition. With popular transformer libraries such as the one by HuggingFace , swapping BERT for one of these variants can be trivially accomplished in many instances. However, experimental results show that improvements on traditional natural language analysis tasks may not translate into improved ranking for information access tasks. Thus, the effectiveness of each BERT variant must be empirically validated. Nevertheless, researchers can potentially "ride the wave" of model advancements at a relatively small cost.
Second, knowledge distillation is a general-purpose approach to controlling effectiveness/efficiency tradeoffs with neural networks. It has previously been demonstrated for traditional natural language analysis tasks, and recent studies have applied the approach to text ranking as well. While knowledge distillation inevitably degrades effectiveness, the potentially large increases in efficiency make the tradeoffs worthwhile under certain operating scenarios. Emerging evidence suggests that the best practice is to distill a large teacher model that has already been trained for ranking into a smaller pretrained student model.
Empirically, BERT has proven to be very effective across a wide range of natural language as well as information access tasks. Combining this fact with the observation that BERT is over-parameterized (for example, Kovaleva et al. [2019] ) leads to the interesting question of whether smaller models might be just as effective, particularly if limited to a specific task such as text ranking. Knowledge distillation from larger BERT models into smaller BERT models represents one approach to answering this question (discussed above), but could we arrive at better effectiveness/efficiency tradeoffs if we redesigned the neural architectures from scratch?
Hofstätter et al.
[2019], tried to answer exactly this question by proposing a text ranking model called the Transformer Kernel (TK) model, which might be characterized as a clean-slate redesign of transformer architectures specifically for text ranking. The only common feature between monoBERT and the TK model is that both use transformers to compute contextual representations of input tokens. Specifically, TK uses separate transformer stacks to compute contextual representations of query and document terms, which are then used to construct a similarity matrix that is consumed by a KNRM variant [Xiong et al., 2017] (covered in Section 1.2.4). Since the contextual representations of terms from the documents can be precomputed and stored, this approach is similar to KNRM in terms of computational costs at inference time (plus the small amount of computation needed to compute a query representation).
The idea of comparing precomputed term embeddings within an interaction-based model has been well explored in the pre-BERT era, with models like POSIT-DRMM [McDonald et al., 2018] , which used RNNs to produce contextual embeddings, but consumed those embeddings with a more complicated architecture involving attention and pooling layers. The main innovation in the Transformer Kernel model is the use of transformers as encoders to produce contextual embeddings, which we know generally perform better than CNNs and RNNs on natural language tasks.
In more detail, given a sequence of term embeddings t 1 , . . . , t n , TK uses a stack of transformer encoder layers to produce a sequence of contextual embeddings T 1 , . . . , T n :
T 1 , . . . , T n = Encoder 3 (Encoder 2 (Encoder 1 (t 1 , . . . , t n )).
This is performed separately for terms from the user's query and terms from the documents in the corpus (the latter, as we note, can be precomputed). The contextual query and document term embeddings are then used to construct a query-document term similarity matrix that is passed to the KNRM component, which produces a relevance score that is used for reranking. Hofstätter et al.
[2019] pointed out that the similarity matrix constitutes an information bottleneck, which provides a straightforward way to analyze the term relationships learned by the transformer stack. The complete TK architecture is shown in Figure 17 In a follow-up, proposed the TK with local attention model (TKL), which replaced the transformer encoder layers' self-attention with local self-attention, meaning that the attention for a distant term (defined as more than 50 tokens away) is always zero and thus does not 
After the contextualization, we match the query sequenceq1:m and document sequenced1:n together in a single match-matrix M 2 R q len ⇥d len with pairwise cosine similarity as interaction extractor:
Mi,j = cos(qi,dj)
Then, we transform each entry in M with a set of RBF-kernels [36] . Each kernel focuses on a specific similarity range with center µk. The size of all ranges is set by . In contrast to Xiong et al.
[36] we do not employ an exact match kernel -as contextualized representations do not produce exact matches. Each kernel results in a matrix K 2 R q len ⇥d len :
Now, we process each kernel matrix in parallel, and we begin by summing the document dimension j for each query term and kernel:
At this point -as shown in Figure 1 -the model flow splits into two paths: log normalization and length normalization. The log normalization applies a logarithm with base b to each query term before summing them up:
To incorporate the notion of different document lengths into the model we enhance the pooling process with document length normalization. We dampen the magnitude of each query term signal by the document length:
Now, the set of kernel scores (one value per kernel) is weighted and summed up with a simple linear layer (Wlog, Wlen) to produce a scalar, for both the log-normalized and length normalized kernels:
Finally, we compute the final score of the query-document pair as a weighted sum of the log-normalized and the length-normalized scores:
We employ kernel-pooling, because it makes inspecting temporary scoring results more feasible compared to pattern based scoring methods (for example PACRR [15] ). Each kernel is applied to the full document and the row-wise and the column-wise summing of the match-matrix allow to inspect individual matches independent from each other.
The main differences of TK in comparison to BERT [24] are:
• TK's contextualization uses fewer and lower dimensional Transformer layers with less attention heads. This makes the query-time inference of TK with 2 layers 40 times faster than BERT-Base with 12 layers. • TK contextualizes query and document sequences independently; each contextualized term is represented by a single vector (available for analysis). BERT operates on a concatenated sequence of the query and the document, entangling the representations in each layer. • The network structure of TK makes it possible to analyze the model for interpretability and further studies. TK has an information bottleneck built in, through which all term information is distilled: the query and document term interactions happen in a single match matrix, containing exactly one cosine similarity value for each term pair. BERT on the other hand has a continuous stream of interactions in each layer and each attention head, making a focused analysis unfeasible.
The differences of TK to previous kernel-pooling methods are:
• KNRM [36] uses only word embeddings, therefore a match does not have context or positional information. • CONV-KNRM [5] uses a local-contextualization with limited positional information in the form of n-gram learning with CNNs. It cross-matches all n-grams in n 2 match matrices, reducing the analyzability. (2) the query-document term similarity matrix is created with pairwise cosine similarities; (3) each kernel creates a new feature matrix and are further manipulated, (4) providing input to a single feedforward layer that computes the final document relevance score.
Method MRR@10 MRR nDCG@10
(1) monoBERT Large (from Table 5 ) 0.372 --(2a) Co-PACRR 0.273 (2b) ConvKNRM 0.277 (2c) FastText + ConvKNRM 0.278 (3a) monoBERT Base 0.376 --(3b) monoBERT Large 0.366 -- need to be computed. TKL additionally used a modified KNRM component that performs pooling over windows of document terms rather than over the entire document.
Extending these idea further, extended TK with the Conformer Kernel (CK) model, which adds an explicit term-matching component based on BM25 and two efficiency improvements: assuming query term independence [Mitra et al., 2019] and replacing the transformer encoder layers with new "conformer" layers. The query term independence assumption is made by applying the encoder layers to only the document (i.e., using non-contextual query term embeddings) and applying KNRM's aggregation to score each query term independently, which are then summed. Similar to TKL's local attention, the proposed conformer layer is a transformer encoder layer in which self-attention is replaced with separable self-attention and a grouped convolution is applied before the attention layer.
The TK, TKL, and CK models are trained from scratch (yes, from scratch) with an embedding layer initialized using context-independent embeddings. The TK and TKL models use GloVe embeddings [Pennington et al., 2014] , and the CK model uses a concatenation of word2vec's "IN" and "OUT" embeddings . This design choice is very much in line with the motivation of rethinking transformers for text ranking from the ground up. However, this also means that these models do not benefit from self-supervised pretraining that is immensely beneficial for BERT (see discussion in Section 3.1). While the models do make use of pretrained embeddings, the transformer layers used to contextualize the embeddings are randomly initialized.
Experiments demonstrating the effectiveness of TK, TKL, and CK are shown in Table 25 , pieced together from a number of sources. Results on the development set of the MS MARCO passage retrieval test collection for TK, rows (4bcd), are taken from , as well their replication of monoBERT baselines, row group (3). For reference, row (1) repeats the effectiveness of monoBERT Large from Table 5 . Although additionally reported results on the MS MARCO document retrieval test collection, we did not include those results here since the TKL and CK papers did not evaluate on that test collection. For TKL, results are copied from on the TREC 2019 Deep Learning Track document retrieval test collection, and for CK, results are copied from for the same test collection. Fortunately, TK model submissions to the TREC 2019 Deep Learning Track provided a bridge to help understand these relationship between these models. From what we can tell, the TK (3 layer, FastText, window pooling) results, row (4a), corresponds to run TUW19-d3-re and the TK (3 layer) results, row (4b), corresponds to runTUW19-d2-re.
To aid in the interpretation of these results, row group (2) provides results from a few pre-BERT interaction-based neural ranking models. Rows (2a) and (2b) are taken directly from for Co-PACRR and ConvKNRM [Dai et al., 2018] . Row (2c) is taken from , which provides more details on the ConvKNRM design. These three results might be characterized as the state of the art in pre-BERT interaction-based neural ranking models, just prior the shift over to transformer-based approaches. We see that the TK model is more effective than these models, but still quite a bit of distance away from monoBERT in terms of effectiveness. Thus, TK can be characterized as a less effective but more efficient transformer-based ranking model, compared to monoBERT. This can be see in Figure 18 , taken from Hofstätter et al. [2020] , which plots the effectiveness/efficiency tradeoff of different neural ranking models. With a latency budget of less than around 200ms, TK is more effective than monoBERT Base and TK represents strictly an improvement over pre-BERT models across all latency budgets.
Interestingly, there is little difference between the two-layer and three-layer TK models, which is consistent with the results presented in the context of distillation above. On the TREC 2019 Deep Learning Track document retrieval test collection, the TK and TKL models perform substantially better than the CK model 99 , though the conformer layers used by CK are more memory-efficient. That is, the design of CK appears to further trade off effectiveness for efficiency. Incorporating an exact matching component consisting of BM25 with learned weights improves the effectiveness of the CK model, but it does not reach the effectiveness of TK or TKL. There does not appear to be much difference in the effectiveness of TK vs. TKL. Unfortunately, it is difficult to quantify the effectiveness/efficiency tradeoffs of TKL and CK compared to TK, as we are not aware of a similar analysis along the lines of Figure 18 .
Takeaway Lessons. What have we learned from these proposed transformer architecture for text ranking? Thus far, the results are a bit mixed.
On the one hand, we believe it is very important for the community to explore a diversity of approaches, and to rethink how we might redesign transformers for text ranking given a blank slate. So far, the TK/TKL/CK models represent the only work we know of that has taken on this challenge, and so it is too early to draw any definitive conclusions from the endeavour. Furthermore, CK represents an exploration of the space between pre-BERT interaction-based neural ranking models and TK, i.e., even more computationally efficient, but also even less effective. There is, in our opinion, an even more interesting tradeoff space between TK and monoBERT. That is, can we give up a bit more of TK's efficiency to close its effectiveness gap with monoBERT?
On the other hand, it is unclear whether the current design of the TK/TKL/CK models can benefit from the massive amounts of self-supervised pretraining that is the hallmark of BERT, and based on the discussion in Section 3.1, is the main source of the big leaps in effectiveness we've witnessed on a variety of NLP tasks. A key question is how these approaches compare to a smaller pretrained BERT model, perhaps one of the distillation approaches discussed in the previous section. In other words, what is more important, model architecture or pretraining? Currently, the jury's still out, but this is definitely an interesting future direction worth exploring.
All of the transformer models that we have discussed so far for text ranking can be characterized as encoder-only architectures. At a high level, these models take as input vector representations derived from a sequence of tokens, and for relevance classification, emit a final probability of relevance (after internal representations have been normalized via a softmax layer). However, the original transformer design by Vaswani et al. [2017] is an encoder-decoder architecture, where an input sequence of tokens is converted into vector representations, passed through transformer encoder layers to compute an internal representation (the encoding phase), which is then used in transformer decoder layers to generate a sequence of tokens (the decoding phase). While the alignment is imperfect, it is helpful characterize previous models in terms of this full encoder-decoder transformer architecture. GPT [Radford et al., 2018] described itself as a transformer decoder, and to fit with this analogy, Raffel et al. [2020] characterized BERT as being an "encoder-only" design.
In NLP, encoder-decoder models are also referred to as sequence-to-sequence models because a sequence of tokens comes in and sequence of tokens comes out. This input-output behavior captures tasks such as machine translation-where the input sequence is in one language and the model output is the input sequence translated into a different language-and abstractive summarization-where the input sequence is a long(er) piece of text and the output sequence comprises a concise summary of the input sequence capturing key content elements.
Until recently, tasks whose output were not comprised of a sequence of tokens, such as the tasks discussed in Section 3.1, were mostly addressed by encoder-only models. These tasks had a natural mapping to the architecture of a model like BERT: Classification tasks over inputs took advantage of the [CLS] representation and [SEP] tokens as delimiters in a straightforward manner. Even though sequence labeling tasks such as named-entity recognition can be conceived as outputting a sequence of tags, a formulation as token-level classification (over the tag space) was more natural since there is a strict one-to-one correspondence between a token and its label (whereas most sequence-to-sequence models do not rigidly enforce this one-to-one correspondence). In this case, the contextual embedding of each token can be used for classification in a straightforward manner. However, with the advent of pretrained sequence-to-sequence models such as T5 [Raffel et al., 2020] , Text-to-Text Transfer Transformer, the community began to explore the use of sequence-to-sequence models for a variety of NLP tasks that were previously addressed with encoder-only models.
The main idea introduced by Raffel et al. [2020] is to cast every natural language processing task as feeding a sequence-to-sequence model some input text and training it to generate some output text. These tasks include those that are more naturally suited for sequence-to-sequence models such as machine translations, as well as tasks for which a sequence-to-sequence formulation might seem a bit "odd', such as detecting if two sentences are paraphrases, detecting if a sentence is grammatical, word sense disambiguation, and sentiment analysis, which are more accurately characterized as either classification or regression tasks. The authors even recast a task like co-reference resolution into this sequence-to-sequence framework.
Like BERT, T5 is first pretrained on a large corpus of diverse texts using an objective similar to masked language modeling in BERT, but adapted for the sequence-to-sequence context. Just like in BERT, these pretrained models (which have also been made publicly available) are then fine-tuned for various downstream tasks using task-specific labeled data, where each task is associated with a specific input template.
These templates provide the cue to tell the model "what to do". For example, to translate a text from English to German, the model is fed the following template:
translate English to German:
where the sentence to be translated replaces [input] and "translate English to German:" is a literal string, which the model learns to associate with a specific task in fine-tuning. In other words, a part of the input sequence consists of a string that informs the model what task it is to perform. To give another example, a classification task such as sentiment analysis (with the SST2 dataset) has the following template: sst2 sentence: [input] (46) where, once again, [input] is replaced with the actual input sentence and "sst2 sentence:" is a literal string indicating the task. For this task, the "ground truth" (i.e., output sequence) for the sequence-to-sequence model is a single token, either "positive" or "negative" (i.e., the literal string). In other words, given training examples processed into the above template, the model is trained to generate either the token "positive" or "negative", corresponding to the prediction. This idea is pushed even further with regression tasks such as Semantic Textual Similarity Benchmark [Cer et al., 2017] , where the target outputs are human-annotated similarity scores between one and five. In this case, the target output is quantized to the nearest tenth, and the model is trained to emit that literal token. Raffel et al. [2020] showed that this "everything as sequence-to-sequence" formulation is not only tenable, but achieves state-of-the-art effectiveness (at the time the model was introduced) on a broad range natural language processing tasks.
Inspired by the success of the sequence-to-sequence formulation, Nogueira et al. [2020] wondered if the T5 model could also be applied to text ranking. It is, however, not entirely straightforward how this can be accomplished. There are a number of possible formulations: As text ranking requires a score for each document to produce a ranked list, T5 could be trained to directly produce scores as strings like in the STS-B task, provided we find the right test collection. Graded relevance judgments might work, but unfortunately most test collections of this type are quite small; the MS MARCO passage retrieval test collection only provides binary relevance.
An alternative would be to encode all the candidate text (from initial retrieval) into a single input template and train the model to select the most relevant ones. However, documents can be long, so this is not feasible given the length limitations of current transformer models; all of our previous discussions in the context of BERT (see Section 3.5) apply here as well. Thus, ranking necessitates multiple inference passes with the model and somehow aggregating the outputs.
Ultimately, to rank with sequence-to-sequence transformers, the solution that Nogueira et al. [2020] devised exploited internal model representations just prior to the generation of an output token for relevance classification. Their model, dubbed "monoT5", uses the following input template:
where [q] and [d] are replaced with the query and document texts, respectively, and the other parts of the template are verbatim string literals. The model is fine-tuned to produce the tokens "true" or "false" depending on whether the document is relevant or not to the query. That is, "true" and "false" are the "target tokens" (i.e., ground truth predictions in the sequence-to-sequence transformation).
At inference time, to compute probabilities for each query-document pair, a softmax is applied only on the logits of the "true" and "false" tokens in the first decoding step. 100 Specifically, the final estimate we are after in relevance classification, P (Relevant = 1|q, d), is computed as the probability assigned to the "true" token normalized in this manner. Similar to monoBERT, monoT5 is deployed as a reranker.
In Section 3.3, we saw that BERT-based rerankers, like BERT-MaxP and CEDR, perform well with (relatively) little training data (e.g., Robust04). With Birch, a relevance matching model was finetuned on out-of-domain data, and in-domain labeled data (Robust04) was only used for tuning a few weights. In Section 3.5.5, we discussed the effectiveness of stage-wise fine-tune using out-of-domain data. These all provide independent evidence that transformers perform well in a few-shot learning setting, and exhibit the ability to transfer learned knowledge (a model of relevance, in this case), across different domains as well as tasks. 101 The logical extreme of learning with limited data (or "few-shot" learning) is zero-shot learning, where a model is directly used without being trained on the task of interest.
With monoT5, Nogueira et al. [2020] explored exactly the zero-shot approach, fine-tuning the model on MS MARCO passage retrieval test collection and directly evaluating on other test collections: Robust04, Core17, and Core18. These results are shown in Table 26 , with the best configuration of Birch, copied from Table 7 . Row group (2) presents results reranking first-stage candidates from BM25 using Anserini, and row group (3) present results reranking first-stage candidates from BM25 + RM3. Each row indicates the size of the T5 model (base, large, and 3B). As expected, effectiveness improves with increased model size, although the differences between the base and large variants are relatively small. The T5-3B model, where "3B" denotes three billion parameters, achieves the highest effectiveness scores on these test collections that we are aware of-which is a remarkable result since T5 was not trained on any of these test collections, but was applied in a zero-shot manner.
Perhaps it is no surprise that larger models improve effectiveness, but Nogueira et al. [2020] argued that the decoder part of the architecture makes important contributions to relevance modeling also. They investigated how the choice of target tokens impact the effectiveness of the model, i.e., the prediction target or the ground truth "output sequence". Perhaps not surprisingly, when the model is fine-tuned with hundreds of thousands of (query, relevant text) pairs, this choice doesn't really matter (i.e., little impact on effectiveness). However, in a low-resource setting (fewer training examples), the authors noticed that the choice of the target token matters quite a bit. In particular, the model does have a preference for learning some target tokens such as "true" vs. "false" (the baseline condition) over others such as "apple" vs. "orange". That is, attempting to coax the model to associate random words with relevance labels is less effective with fewer training examples, which suggests that the model is leveraging the decoder part of the network to assist in building a relevance matching model. Exactly how, Nogueira et al. [2020] provided no explanation. This result raises the interesting question whether there are "optimal" input and output templates to sequence-to-sequence models. Furthermore, are there ways to automatically find templates to help the model learn more quickly, using fewer training examples? These and related questions are also being investigated by other researchers , Schick and Schütze [2020] and together these studies might lead to a better understanding of the inner working of large sequence-to-sequence models.
Takeaway Lessons. In a way, text ranking with sequence-to-sequence models represents the philosophical opposite of the motivation behind the TK, TKL, and CK models. Instead of rethinking the design of transformer architectures from the ground up, Nogueira et al. [2019b] opted for the "more pretraining, bigger models" approach, taking advantage of broader trends in NLP; GPT-3 [Brown et al., 2020] is perhaps currently the most extreme expression of this philosophy. On the one hand, the "results speak for themselves", as the monoT5 results represent the highest reported effectiveness on multiple test collections that we are aware of. On the other hand, these numbers alone are unsatisfying.
There are still open questions on how exactly monoT5 achieves these numbers: Is it just a larger model, or are there useful signals in the decoder for ranking? Although Nogueira et al. [2019b] had begun to explore these questions, they offered no conclusive answers.
We have arrived at the research frontier of text ranking using transformers in the context of reranking approaches. Where do we go from here? Direct ranking with continuous dense representations is an emerging area that we cover in Section 4, but beyond that lies unexplored ground. There are a number of promising future paths, which we return to discuss in Section 5.
In truth, every application is domain specific: typical corpora that are used by researchers, such as books, Wikipedia, web pages, newswire articles, etc. do, in fact, represent different domains. What the research community means by domain-specific applications usually refers to domains that are very different from the corpora that most researchers typically use (e.g., to pretrain models). For the purposes of this discussion, we will use domains and corpora interchangeably, as the most accurate way to operationalize a particular domain is to simply refer to the texts that comprise that domain.
Perhaps the biggest challenge with domain-specific applications is vocabulary differences between the target corpora (we use this term to refer to the specific domain we care about) and the corpora used to pretrain models. From the perspective of information access, this will likely result in mismatches between the terms used in queries and terms used in the target texts. For instance, the term "WordPiece" appears four times in the BERT paper but the term "subword" does not appear even once, though "WordPiece" is clearly a type of subword tokenization. A model that does not know the relationship between "WordPiece" and "subword" would have problems in answering the question "what type of subword does BERT use?"
Beyond vocabulary differences, of course, texts in different domains vary in their usages of language at the syntactic, semantic, and even discourse levels. While these differences are undoubtedly important, especially for human readers, we argue that the challenges begin with vocabulary differences. Pretrained models today acquire their knowledge of language implicitly via simple objectives (e.g., masked language model), and if a term rarely occurs, there is little that can be learned. Subword tokenization mitigates the problem as the representation of compound words such as "firefighter" can be learned by reusing representations of the words "fire" and "fighter". However, in many cases, the subwords are only loosely related to the meaning of the full word. For instance, it is harder for a model to learn that a "coronavirus" causes a respiratory disease if it only learned about the words "corona" and "virus", since "corona" exists in diverse contexts unrelated to health, and "virus" occurs with many different types of diseases. Hence, the representation of the full word still need to be learned from its occurrence in text. Not surprisingly, researchers have specifically built adaptations of BERT that attempt to address these vocabulary issues, for example, SciBERT [Beltagy et al., 2019] and BioBERT [Lee et al., 2020b] , but the most effective pretraining process remains an open research problem [Gu et al., 2020] .
In this section, we describe some recent applications of transformer-based models in domain-specific applications, focusing in particular on two domains: the medical and biomedical domain (Section 3.7.1) and the legal domain (Section 3.7.2). In no way do we intend for each discussion to be comprehensive, as a proper treatment of each domain would demand a book in itself [Benjamins et al., 2005 , Hersh, 2020 . Instead, our focus is on recent advances with BERT and other transformer architectures, highlighting important findings and trends. Note that even though we limit our discussion to information access, domain-specific applications are typically paired with domain-specific tasks. As will become clear, the characteristics of "search" become different in the context of searching scientific articles or legal documents.
As we have already discussed at the beginning of the book (Section 1.2), many of the earliest information retrieval systems were motivated by the desire to tame the growth of knowledge accumulated in the scientific literature. While information access technologies can be targeted at any scientific discipline, (clinical) medicine and biomedicine (and more broadly, the life sciences) has attracted the most attention, for several reasons. Successful applications can potentially have a high impact 102 and the importance of health care in modern societies (economically, politically, etc.) translates into substantial investments of resources in research from governments and other organizations.
Even limiting the scope of discussion to information access, it is important to understand that there are many "flavors" of search in the medical and biomedical domain. As we would expect, there are research-oriented questions, for example, "What is the link between the DRD4 allele to alcoholism?" 103 Systems for addressing such information needs might be targeted to researchers across the basic to applied spectrum.
Another major application is decision support for clinical care, under the general paradigm known as evidence-based medicine (EBM) [Straus et al., 2018] , which is a fancy name for the common sensical idea that physicians make "conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients" [Sackett et al., 1996] . To provide an over-simplified example, when deciding between two drugs to prescribe, the physician's decision should be guided by clinical trials that have compared the efficacy of the two drugs in a randomized, controlled setting, targeting populations matching the patient in question (as opposed to, say, the marketing literature of competing pharmaceutical companies). Since the results of these trials are primarily published in peer-reviewed journal articles, the practice of EBM requires consulting (i.e., searching) the scientific literature and properly interpreting the findings reported in those trials. Since it is unlikely that a practicing physician would have the time to do this for every patient and for routine situations, clinical care guidelines have emerged to capture best practices (i.e., when faced with X, generally do Y unless Z, etc.). These guidelines, in turn, are created and periodically updated by experienced and highly-qualified teams of physicians who perform systematic reviews (also called meta-analyses) of the literature to interpret and synthesize the available clinical evidence; the Cochrane Database of Systematic Reviews is a prominent example.
The common feature of both types of information needs (i.e., those that are research in nature and those that are clinical in nature) is the importance of high recall. This contrasts sharply with typical requirements in web search, where users only need a few relevant documents. In the medical and biomedical domains, users desire as much relevant material as possible-all relevant articles, if possible. This is typically operationalized in the choice of metrics: early-precision metrics such as nDCG@10 do not accurately model the needs of information seekers with research and clinical questions.
It should be no surprise that the information retrieval community has been working with medical and biomedical texts as soon as machine-readable corpora became available. For example, much of the "indexing wars" in the 1960s and 1970s-the debate between human-derived and automaticallygenerated index terms-were fought in the context of scientific articles in the life sciences [Salton, 1972] (see Section 1.2.1). TREC tracks have been organized in this domain as well: the Genomics Track, which ran from 2003 [Hersh and Bhupatiraju, 2003 ] to 2007 [Hersh et al., 2007] , represented information needs closer to those of researchers. The Precision Medicine Track, which began in TREC 2017 [Roberts et al., 2017] and continues to this day, focuses on clinical information seeking scenarios in the context of evidence-based medicine. Beyond TREC, another notable effort is the BioASQ workshop series, which has been organizing evaluation challenges on biomedical semantic indexing and question answering since 2013 [Tsatsaronis et al., 2015] . Among the earliest known applications of BERT to information access in this domain is the work of Pappas et al. [2019] , who examined document and snippet retrieval in the context of BioASQ.
In 2020, the most significant event that has disrupted all aspects of life worldwide is, of course, the COVID-19 pandemic. Improved information access capabilities has a role to play in the fight against this disease by providing stakeholders with high-quality information from the emerging scientific literature, to inform evidence-based decision making and to support insight generation. Examples might include public health officials assessing the efficacy of population-level interventions such as mask ordinances, clinicians conducting meta-analyses to update care guidelines based on emerging clinical studies, and virologists probing the genetic structure of the virus to develop vaccines. As our knowledge of COVID-19 is rapidly evolving and results of new studies are becoming available at a rapid pace, there is a constant need to reassess current practices against the latest evidence, necessitating high-quality information access tools to sort through the literature.
From the research perspective, the pre-requisite to developing and evaluating these capabilities is a publicly accessible corpus that scientists can work with. As a response to this need, on March 13, 2020, the Allen Institute for AI (AI2) released the COVID-19 Open Research Dataset (CORD-19) , with regular updates since its initial release (first weekly, then daily). CORD-19 is a curated corpus of scientific articles about COVID-19 and coronavirus-related research (for example, SARS and MERS). Most articles in the corpus contain the full text, while only abstracts and metadata are available for the rest. These articles are gathered from a variety of sources, including PubMed, a curated list of articles from the WHO, as well as preprints from arXiv, bioRxiv, and medRxiv. The goal of the effort is "to mobilize researchers to apply recent advances in natural language processing to generate new insights in support of the fight against this infectious disease." Indeed, CORD-19 has catalyzed much research activity, not only in information access, but also NLP, data mining, and related disciplines.
The NIST-organized TREC-COVID challenge , 104 which began in mid-April 2020 and lasted until late-August 2020, brought TREC-style evaluations to the CORD-19 corpus. The stated goal of the effort was to provide "an opportunity for researchers to study methods for quickly standing up information access systems, both in response to the current pandemic and to prepare for similar future events." Due to the rapid growth of the scientific literature, the TREC-COVID challenge was organized into a series of "rounds", each of which used a particular snapshot of the CORD-19 corpus.
The topics for the evaluation comprised information needs that were primarily clinical in nature (e.g., "Are patients taking Angiotensin-converting enzyme inhibitors (ACE) at increased risk for COVID-19?"), although some topics had broader public health implications (e.g., "What are the best masks for preventing infection by Covid-19?"). However, due to the rapid nature in which evaluation resources were organized, key aspects of the underlying information access challenges were not adequately modeled-for example, relevance judgments did not take into account the quality of the retrieved evidence (e.g., whether the article described results from a randomized controlled trial). Also, since it was recognized at the outset that proper evaluation of recall was unlikely to be feasible, TREC-COVID primarily focused on early-precision metrics such as nDCG.
From a methodological perspective, TREC-COVID implemented a few interesting features that make it stand out from other TREC evaluations. Each round contained both topics that were persistent (i.e., carried over from previous rounds) as well as new topics-the idea was to both update the state of knowledge with respect to existing information needs based on new evidence as well as to address emerging information needs. The assessment approach followed a standard pooling strategy, but once an article was assessed, its judgment was never revised (even if contrary evidence later emerged). To avoid duplicate effort, the evaluation adopted a residual collection methodology, where previously judged articles were automatically removed from consideration. Thus, each round only considered articles that had not been examined before by a human assessor (on a per-topic basis); these were either newly published articles or existing articles that had not previously been submitted as part of a run (and hence never entered the judgment pool in a previous round). Round 1 began with 30 topics, and each subsequent round introduced five additional topics, for a total of 50 topics in round 5.
This evaluation methodology had some interesting implications. On the one hand, each TREC-COVID round stands separately as a "mini-evaluation", in the sense that scores across rounds are not comparable: both the corpora and the topics are different. On the other hand, the partial overlaps in both topics and corpora across rounds provide connections that link the series of rounds. In particular, for the persistent information needs, relevance judgments from previous rounds can be exploited to improve the effectiveness of systems in future rounds on the same topic. Runs that took advantage of these relevance judgments are known as "feedback" runs, in contrast to "automatic" runs that did not. A third category, "manual" runs, can involve any human input, such as human judgments or manual query formulation.
Overall, the TREC-COVID challenge was a success in terms of participation and in attracting substantial attention from the research community. The first round had over 50 participating teams from around the world, and although the participants dwindled somewhat as the rounds progressed, round 5 still had close to 30 participating teams. For reference, a typical "successful track" an TREC might draw around 20 participating teams, such as the TREC 2019 Conversational Assistance Track (CAsT) Track.
From the perspective of this survey, the TREC-COVID challenge is of particular interest because it represented the first large-scale evaluation of domain-specific information access capabilities following the introduction of BERT-based models for text ranking. As expected, the evaluation showcased a variety of transformer-based models. Furthermore, participation from teams that did not base their runs on transformers produced a natural contrast in experimental approaches. Since all participants began with no in-domain relevance judgments, the evaluation provided an interesting case study in rapid domain adaption. However, the multi-round setup allowed teams to improve their systems based on previous results, train their models using newly available relevance judgments, and generally improve their techniques based on accumulated experience. In terms of approaches based on supervised machine learning, the biggest challenge was the relative paucity of domain-specific labeled training data: on a per-topic basis, there were only a few hundred total judgments (both positive and negative) per round. Nevertheless, the evaluation setup introduced a realistic scenario to challenge the application of transformer architectures for ranking.
Overall, the evaluation exercise was a driver of rapid progress in the field. The nature of the pandemic meant that research, system development, and evaluation efforts were intense and compressed into a short time span, thus leading to rapid advances that otherwise might have taken several TREC cycles to play out. Similarly, innovations diffused from group to group much faster than they would have had under normal circumstances.
It should come as no surprise that the top scoring "feedback" and "automatic" runs in the TREC-COVID challenge were based on BERT and other transformer architectures. Beyond this, however, we attempt to summarize some of the important lessons learned below. As the evaluation only concluded in August 2020, the community has not had sufficient time to digest the wealth of data that the evaluation effort has generated. To date, there are relatively few publications (pre-prints or otherwise) that describe the approaches of the participants. In many cases, the only source for understanding the techniques used in a run is the paragraph-length description associated with that submission, found on the NIST website for the evaluation. 105 These descriptions, unfortunately, are often lacking in details. Nevertheless, some high level findings have emerged:
Ensembles and fusion techniques work well. Many teams submitted runs that incorporated the output of different techniques. Some of these "base techniques" were relatively simple, for example, exact match scoring against different representations of the articles (e.g., abstracts, full texts, and paragraphs from the full text). Other sources of fusion involved variants of BERT-based models, or transformer-based rerankers applied to different first-stage retrieval approaches.
Simple fusion techniques such as reciprocal rank fusion [Cormack et al., 2009] or linear combinations [Vogt and Cottrell, 1999] were effective and robust, with few or no "knobs" to tune-and therefore far less reliant on training data than supervised machine learning techniques. In the earlier rounds, this was a distinct advantage as all the teams were equally inexperienced in working with the CORD-19 corpus. In the first round, for example, the best "automatic" run was submitted by the "sabir" team, who combined evidence from bag-of-words vector-space retrieval against abstracts and full text using linear combination. Even in the later rounds, ensembles and fusions techniques still provided a boost over individual transformer-based ranking models. Some sort of fusion technique was adopted by nearly all of the top-scoring runs across all the rounds. While the effectiveness of ensemble and fusion techniques is not a new observation in search (above references, as well as [Bartell et al., 1994, Montague and Aslam, 2002] ), seeing that these general findings are reproduced in a new context enhances our confidence in and understanding of the underlying techniques.
Simple domain adaptation techniques work well with transformer architectures. Even prior to the COVID-19 pandemic, NLP researchers had already developed (and made publicly available) different versions of BERT that were pretrained on the scientific literature. SciBERT [Beltagy et al., 2019] and BioBERT [Lee et al., 2020b] are two well-known examples, and these models served as the starting points for many techniques used in the evaluation.
In terms of fine-tuning BERT-based ranking models specifically for TREC-COVID, one noteworthy innovation was the automatic creation of (pseudo) in-domain training data from a general-domain dataset proposed by . The idea consisted of filtering from the MS MARCO passage retrieval test collection and retaining only queries (and their associated relevant passages) that contain at least one term in the MedSyn lexicon, which includes layperson terminology for various medical conditions [Yates and Goharian, 2013] . That is, used this simple technique to create a "medical subset" of the MS MARCO passage retrieval test collection, which was then used to fine-tune a monoBERT model based on SciBERT [Beltagy et al., 2019] . In the first round, this run was the second highest scoring "automatic" run, but alas, it was still not as effective the simple bag-of-words fusion run from the "sabir" team mentioned above. Domain adaptation using data selection tricks to choose portions of a larger dataset that are closer to the target test data have been explored previously in the context of machine translation [Axelrod et al., 2011] , but it was quite surprising that simple keyword-based filtered performed so well. This innovation was simple and effective, thus quickly adopted by the other participants in subsequent rounds.
Another quite interesting method to create in-domain labeled data was used by the team "unique_ptr". Their approach consisted of generating queries from CORD-19 articles using a model similar to doc2query and then training a reranker on these pseudo query-relevant document pairs [Ma et al., 2020] . The "unique_ptr" team was responsible for the highest "feedback" run (and top-scoring run overall) in rounds 4 and 5, and these runs incorporated this data generation approach.
As the rounds progressed, transformer-based ranking models generally became more effective relative to other types of runs, as researchers began to gain more experience with the topics and the corpora. Results from the best automatic runs in rounds 4 and 5 used some of the techniques we have already discussed in this survey. The "covidex" team deployed an architecture comprising doc2query (using T5) for document expansion (Section 3.5.2) and a reranking pipeline comprising monoT5 and duoT5, which were sequence-to-sequence variants of the monoBERT/duoBERT combination discussed in (Section 3.4.1). Interestingly, their approach can be considered zero shot, since each stage of the pipeline was trained with only the MS MARCO passage retrieval test collection.
Learning with limited data remains a weakness with transformers architectures. Recall that "feedback" runs were allowed to take advantage of relevance judgments from previous rounds, whereas "automatic' runs could not. In general (especially in the later rounds), we see that automatic runs based on transformer architectures outperformed non-transformers runs by a large margin, whereas in many cases feedback runs based on transformer architecture barely beat their non-transformer competition. Simple relevance feedback techniques, for example, were quite competitive compared to transformer-based approaches. For example, in round 2, a "feedback" run submission by the "UIowaS" team, which can be characterized as off-the-shelf relevance feedback, reported the third highest score in that run category. Although two BERT-based "feedback" runs from the "mpiid5" team outperformed this relevance feedback approach, the margins were quite slim.
As another example, the "covidex" team implemented an approach that treated relevance feedback as a document classification problem using simple linear classifiers [Yu et al., 2019 , Lin, 2019b . In both rounds 4 and 5, it was only narrowly beaten by a far more complicated BERT-based ensemble setup by the team "unique_ptr" (that included the synthetic data generation approaches discussed above). This means that, on the whole, researchers have yet to figure out how to effectively exploit the relatively small amount of relevance judgments that are available. How to fine-tune BERT and other transformer-based models with limited data remains an open question, not only for text ranking, but across other NLP tasks as well [Zhang et al., 2020e, Lee et al., 2020a . The evaluation exposed weaknesses in the evaluation methodology itself. One nice feature of TREC is that not only does it a provide a forum for assessing systems and techniques, but it also provides opportunities to evaluate benchmarking methodologies themselves. The design of TREC-COVID was dictated by many practical constraints and organized in a short amount of time; typically, organizers have an entire year to prepare for a new track. There were a number of known limitations at the outset, already discussed above: not explicitly considering the quality of the evidence during assessment and the use of early-precision metrics such as nDCG@10.
The practical (already known at the outset) consequence of the second choice is that participants optimized for early-precision metrics, which is contrary to the demands of a real decision support system, where it is crucial that systems retrieve as much relevant material as possible. This had an unintended consequence as well-from the perspective of early-precision metrics, the information needs were too easy! In the final round, the best "feedback" system received an nDCG@20 of 0.850. In terms of a metric like P@10, the best systems were already nearly "perfect"! Of course, MAP was much lower, but whether that metric is fair and reliable (in the sense of the discussions in Section 2.1) remains to be seen. In fairness to the organizers, however, predicting the difficulty of a topic is already a major challenge (see, for example, Hauff et al. [2010] ); doing so against an evolving corpus basically required the impossible feat of predicting the future! Another unanticipated issue was the existence of near-duplicate articles in the corpus-articles with very similar content, but different ids. These arose when, for example, a preprint was submitted to a peer-reviewed venue and later published as a refereed article. Since deduplication for the residual test collection methodology was performed by unique ids, in many cases essentially the same article was retrieved more than once and thus entered the judgment pool multiple times. While researchers have previously studied the impact of near-duplicate documents, e.g., [Bernstein and Zobel, 2005, Fröbe et al., 2020] , a thorough analysis has not been applied to TREC-COVID.
Takeaway Lessons. We have shared some preliminary lessons from the TREC-COVID challenge here, but no doubt these findings will be augmented and refined in the coming months as researchers describe their approaches in more detail and conduct follow-up experiments. At a high level, though, the evaluation has affirmed the overall effectiveness of transformer-based ranking models, Nevertheless, results identify learning under limited data conditions as a weakness for transformer architectures and unresolved issues with respect to the evaluation methodology itself, both requiring additional future work.
Like applications of information retrieval techniques to the medical and biomedical domain, the first search systems for the legal texts date to at least the 1960s [Robins, 1967] , with retrieval techniques based on neural networks appearing as early as the late 1980s [Belew, 1987] . AI techniques have been applied to the legal domain since the 1980s [Susskind, 1986] (expert systems at the time). The conference series "Artificial Intelligence and Law" began in 1987 and continues to this day. While the community is relatively small, there has been active research at the intersection of computer science and the law for decades. See Zhong et al. [2020] for a recent overview of how NLP can benefit the legal system, which also provides a summary of historical developments.
Broadly speaking, two main applications have emerged. The first is known as "e-discovery", which is the retrieval of electronic business records (for example, email) for use as evidence in civil litigation. The TREC Legal Track, which ran from 2006 [Baron et al., 2006 ] to 2011 [Grossman et al., 2011] , was organized to tackle this challenge. As with all TREC efforts, the track sought to foster the growth of a research community around this topic as well as to build test collections, baselines, and other resources. The focus of this track was high-recall retrieval, where parties desire as much of the relevant material as possible. This requirement is similar to systematic reviews in the medical context, and as already discussed, differs substantially from the needs of, say, casual web searchers. These evaluations mostly predated the advent of neural techniques, although the test collections that have been created can potentially provide resources for evaluating the effectiveness of transformer architectures. However, based on experience from TREC-COVID, it is not clear that in the high recall scenario, transformers are obviously better than much simpler classification approaches such as linear models. Some evidence for this comes from the "feedback" runs from TREC-COVID, where the gap between transformer-based methods and much simpler techniques is quite modest.
Note that although the TREC Legal Track modeled a real-world task faced by lawyers, the corpus comprised primarily of business documents (for example, emails), not legal documents. However, the increasing amount of legal documents available digitally has enabled development of the second main application of information access capabilities, targeted at case law. Such capabilities are particularly pertinent in common law countries such as Australia, Canada, India, New Zealand, United Kingdom, and United States, where judicial decisions are made based on similar and representative cases in the past, i.e., precedents. To settle a dispute, a common law court looks to previous cases and applies the principles from those cases to arrive at a resolution; typically, courts are bound to follow the reasoning used in those precedents. The task of finding similar cases is fundamentally a search problem. A typical usage scenario of a legal information retrieval system involves a user (typically, a legal professional) who needs to find information relevant to a case of interest, often called "base case". The corpus is typically comprised of case law (including both briefs and opinions) as well as secondary sources (e.g., law review articles).
While certainly not as plentiful as newswire test collections, there are resources available for searching legal documents. A recently built test collection is CaseLaw [Locke and Zuccon, 2018] , which comprises approximately 4M judicial decisions in the United States (serving as the corpora), 12 base cases (serving as the topics), and 2572 relevance assessments made by legal professionals. A similar dataset is CM [Xiao et al., 2018] , which contains 8,964 triples (A, B, C) consisting of legal documents, where the annotation is whether B or C is more similar to A. Zhong et al. [2020] showed that a BERT-based model performs better a tf-idf similarity model in this task, but is slightly less effective than an attention-based convolutional neural network [Yin et al., 2016] . There are relatively few transform-based approaches to legal information retrieval, and BERT-based models have only begun to be adopted in legal NLP applications more broadly [Yeung, 2019 , Elwany et al., 2019 .
Recognizing the importance of community-wide evaluations in driving progress, there have been recent efforts specifically focused on the legal domain. The most prominent of these is the Competition on Legal Information Extraction/Entailment (COLIEE) [Rabelo et al., 2019b , Kano et al., 2018 . The latest edition of the evaluation (2019) considered both NLP and IR tasks:
• Task 1: legal case retrieval, where the goal is to find relevant cases (called "noticed cases") to a base case among a collection of cases. • Task 2: legal case entailment, where the goal is to identify paragraphs from existing cases that entail the decision of a base case. • Task 3: statute law retrieval, where given a yes/no question from Japanese legal bar exams, the task is to retrieve Japanese civil code articles S that are relevant. • Task 4: statute law entailment, where given a question from task 3 and an article, the system needs to decide if the article entails or not the question.
We see that there are formulations of tasks in the legal domain that are quite analogous to common NLP and IR tasks in more familiar domains. Moreover, there appears to be interest and early work in applying neural networks (more broadly) and transformer architectures (more specifically) to tackling these challenges. Among the seven participating teams for task 1, the submissions of the top two teams were based on neural networks and the second best used a BERT-based reranker similar to monoBERT [Rossi and Kanoulas, 2019] . In task 2, the best submission was a BERT-based model [Rabelo et al., 2019a] . However, more explorations are clearly needed: for task 3, of the seven participating teams, none used BERT-based models and the best submissions were based on exact match techniques [Kim et al., 2019] . In task 4, there were also seven participating teams, and the best submission was based on rules [Kim et al., 2019] .
Takeaway Lessons. While there is substantially less activity overall in the application of the latest neural techniques, compared to the medical and biomedical domain, the legal domain remains a fertile ground for future innovations and provides a potentially high-impact application to enhance the functioning of modern societies.
Arguably, the single biggest benefit brought about by modern deep learning techniques to text ranking is the move away from sparse signals, mostly limited to exact matches, to continuous representations that are able to capture semantic matches to better model relevance (see Section 1.2). The potential of dense representations for analyzing natural language was first demonstrated with word embeddings on word analogy tasks, which is generally viewed as the beginning of the "neural revolution" in natural language processing. However, as soon as we try to build continuous representations for any larger spans of text (phrases, sentences, paragraphs, and documents), many of the same issues that arise in text ranking come into focus. Here, as we will see, there is a close relationship between notions of relevance from information retrieval and notions of semantic similarity and paraphrase from natural language processing.
Initial attempts at deriving representations for sequences of tokens were based on simple techniques such as averaging the representations of the individual tokens (i.e., word embeddings), but researchers quickly realized the limitations of these techniques and began to investigate more sophisticated approaches, including with neural networks. The challenge of building representations of token sequences from the representations of the individual tokens is sometimes called the composition problem, since semantics is generally assumed to be compositional (i.e., the meaning of a sentence is derived from the meaning of its constituent words). A surprising finding from early work is that simple compositions of token representations (e.g., averaging) are quite competitive with much more complex techniques.
The focus of this section is the application of transformers to generate representations of texts that are suitable for ranking, in both the unsupervised and supervised setting; this is often called representation learning. Tackling this challenge, naturally, requires addressing the composition problem, since the input to transformers remains vector representations (i.e., embeddings) of the individual tokens.
We begin with a more precise formulation of what we mean by text ranking using dense representations, clearly establishing the connection between relevance and semantic similarity (as well as related concepts). In particular, while we adopt a ranking perspective, the core challenge remains the problem of estimating the relation between two pieces of texts.
In the same way that keyword search requires invert indexes and query evaluation infrastructure to support top k ranking using sparse exact match features, top k ranking in terms of simple vector comparison operations such as cosine similarity on dense representations requires dedicated infrastructure as well. This problem, known as nearest neighbor search, is discussed in Section 4.2.
As with neural ranking models, it is helpful to discuss historical developments in terms of "pre-BERT" and "post-BERT" techniques: Section 4.3 overviews ranking based on dense representations prior to BERT. We can clearly see connections from recent work to similar ideas that have been explored for many years, the main difference being the type of neural model applied. Our survey of ranking techniques using transformer-based dense representations is divided into two parts: Section 4.4 covers ranking techniques based on simple comparison functions, while Section 4.5 discusses approaches based on more complex comparison functions.
We begin by more precisely defining the family of techniques covered in this section. Because text ranking with dense representations is an emerging area of research, the literature has not yet converged on consistent terminology. In this survey, we try to synthesize existing work and harmonize different definitions without unnecessarily introducing new terms.
The core problem of text ranking remains the same as the setup introduced in Section 2: We assume the existence of a corpus C = {d i } comprised of an arbitrary number of texts. Given a query q, the task is to generate a top k ranking of texts from C that maximizes some metric of quality. In the multi-stage ranking architectures covered in Section 3, this is accomplished by keyword search as the initial retrieval stage (i.e., retrieval is based on sparse vector representations derived from bags of words), followed by one or more rerankers (based on BERT or some other transformer architecture operating on dense representations).
Here, in contrast, the setup is different. We would like to learn some transformation η : [t 1 ...t n ] → R n on queries and texts, i.e., η(q) and η(d), that converts sequences of tokens (texts) into vectors. Typically, η takes as input the vector representation (i.e., word embeddings) of the individual tokens in the sequence. The goal of this transformation is to recast the ranking problem in terms of a simple similarity function φ, typically, cosine similarity. That is, we wish to find the top k vectors with the highest similarity, or equivalently, the smallest distance. This is also a top k retrieval problem, but performed with different infrastructure (see Section 4.2).
Specifically, applied to text ranking for search, we wish to estimate the following:
that is, the relevance of a text with respect to a query. 106 Since there is no currently agreed upon symbol for this transformation (also called a representation function) in the literature, we introduce the symbol η (i.e., eta) as a mnemonic for "encoder". We use this term throughput this section since it appropriately evokes the notion of feeding the input sequence into a deep neural network. Encoders for queries and texts could either be the same or each could use a separate encoder; we discuss this design choice in more detail below.
The output of the encoder η is a dense representation. One intuitive way to think about these representations is "like word embeddings, but for sequences of tokens". In fact, most designs take as input a sequence of word embeddings that correspond to each input token. These representations are dense in the commonly understood sense, typically having hundreds of dimensions, as opposed to sparse representations where the number of dimensions is equal to the vocabulary size, with most the elements being zero. Thus, dense representations for ranking establishes a poignant contrast to sparse representations for ranking, which can be used to describe BM25-weighted document vectors based on bags of words.
What about the similarity function? Generally, φ is assumed to be a symmetric function, i.e., φ(u, v) = φ (v, u) . Furthermore, φ should be "fast to compute". There is, unfortunately, no precise, widely agreed upon definition of what this means, except by illustration. Most commonly, φ is defined to be cosine similarity between the representation vectors (or inner products, where the only difference is normalization); other metrics such as such as L 1 are also used as well. However, it is understood that φ cannot be a deep neural network-otherwise, we could just define φ to be inference by BERT, and we're back to something like the monoBERT model again. Nevertheless, as we will discuss, there are interesting options that occupy the middle ground.
Thus, ranking techniques for dense representations need to address two challenges:
• the representation problem, or the design of the encoder η, to accurately capture the semantics of the input texts for the purposes of ranking; and, • the comparison problem, or the design of φ, which involves a balance between what can be easily computed and what is necessary to capture salient differences between the encoded representations.
As we'll discuss in Section 4.3, these challenges predate BERT, although transformers broaden the design space of η and φ.
Note that the complete model comprised of η and φ can be unsupervised or supervised. An encoder that is "unsupervised" means that the model has never been exposed to labeled training data corresponding to the target task. However, the transformers that comprise η are almost always pretrained (after all, that's the entire point) and in the pretraining process, the model may have been exposed to the target corpus. In contrast, in a supervised model, η (and φ in some cases as well) are trained (or fine-tuned, to be more accurate) on labeled training data corresponding to the target task, e.g., pairs of (query, relevant text) for relevance ranking. The output of η in the supervised scenario is called a learned representation, and the problem formulation is an instance of representation learning.
Another way to think about dense representations for ranking is to recall the evolution of broad classes of neural ranking models, captured by Figure 9 and repeated here as Figure 19 . Dense representations for ranking are most similar to representation-based approaches, except that more Figure 19 : The evolution of neural models for text ranking, copied from Figure 9 : representation-based approaches (left), interaction-based approaches (middle), and BERT (right). Dense representations for ranking are most similar to representation-based approaches, except that more powerful transformerbased encoders are used to model queries and texts from the corpus.
powerful transformer-based encoders are used to model queries and texts from the corpus. In previous models, the "arms" of the network that generate representations for the query and the text from the corpus are based on CNNs and RNNs. Today, these have been replaced with BERT and other transformer architectures. For the choice of the comparison function, pre-BERT representation-based neural ranking models adopt simple designs for φ, for example, cosine similarity. With transformerbased representations, such simple comparison functions remain applicable. However, researchers have become to explore more complex designs for φ, as we will see in Section 4.5.
What are the motivations for exploring this formulation of the text ranking problem? We can point to two main reasons:
• BERT inference is slow. This fact, as well as potential solutions, was detailed in Section 3.6. The formulation of text ranking in terms of φ(η(q), η(d)) has two key properties: First, note that η(d) is not dependent on queries. This means that text representations can be precomputed and stored, thus pushing potentially expensive neural network inference into a preprocessing stage-similar to doc2query, DeepCT, and HDCT (see Section 3.5). Although η(q) needs to be computed at query time, it requires only a single inference, and typically over a sequence of tokens that is much shorter than the average length of texts. Second, the similarity function φ is fast by construction and ranking in terms of φ over a large (precomputed) collection of dense vectors is typically amenable to solutions based on nearest neighbor search (see Section 4.2). • Multi-stage ranking architectures are inelegant. Initial candidate retrieval is based on keyword search operating on sparse (bag of words) representations, while all subsequent neural ranking models operate on dense representations. This has a number of consequences, the most important of which is the inability to perform end-to-end training. In practice, the different stages in the pipeline must be optimized separately. Typically, first-stage retrieval is optimized for recall, to provide the richest set of candidates for the reranker to work with. However, increased recall in candidate generation provides no guarantee for higher end-to-end effectiveness. One reason for this is that there is often a mismatch between the data used to train the reranker (a static data set, such as the MS MARCO passage retrieval test collection) and the text that it actually sees at inference time (the output of BM25 ranking). Although this mismatch can be mitigated by data augmentation and sampling tricks, they are heuristic at best. Alternatively, if the text ranking problem can be boiled down to the comparison function φ, we would no longer need multi-stage ranking architectures. This is exactly the promise of representation learning: that we can learn neural networks that generate representations directly optimized in terms of similarity according to φ.
Before describing ranking techniques for dense representations, it makes sense to discuss some highlevel modeling choices. The ranking problem we have defined in Eq. (48) shares many similarities with, but is nevertheless distinct from, a number of natural language analysis tasks that are functions of two input sequences:
• Semantic equivalence. Research papers are often imprecise in claiming to work on computing "semantic similarity" between two texts, as semantic similarity is vague notion. 107 Most research, in fact, use semantic similarity as a shorthand to refer to a series of tasks known as the Semantic Textual Similarity (STS) tasks [Agirre et al., 2012 , Cer et al., 2017 . Thus, semantic similarity is operationally defined by the annotation guidelines of those tasks, which fall around the notion of semantic equivalence, i.e., "Do these two sentences mean the same thing?" While these concepts are notoriously hard to pin down, it is clear that the task organizers have carefully thought through and struggled with the associated challenges; see for example, Agirre et al. [2012] . Ultimately, these researchers have built a series of datasets that reasonably capture operational definitions amenable to computational modeling. 108
• Paraphrase. Intuitively, paraphrase can be understood as synonymy, but at level of token sequences. For example, "John sold the violin to Mary" and "Mary bought the violin from John" are paraphrases, but "Mary sold the violin to John" is not a paraphrase of either. We might formalize these intuitions in terms of substitutability, i.e., two texts (phrases, sentence, etc.) are paraphrases if one can be substituted for another without significantly altering the meaning. From this, it is possible to build computational models that classifies text pairs as either being paraphrases or not. 109
• Entailment. The notion of entailment is formalized in terms of truth values: a text t entails another text h if, typically, a human reading t would infer that h is most likely true [Giampiccolo et al., 2007] . Thus, "John sold the violin to Mary" entails "Mary now owns the Violin". Typically, entailment tasks involve a three-way classification of "entailment", "contradiction", or "neutral" (i.e., neither). Building on the above example, "John then took the violin home." would contradict "John sold the violin to Mary", and "Jack plays the violin." would be considered "neutral" since the original sentence tells us nothing about Jack.
Thus, relevance, semantic equivalence, paraphrase, entailment are all "similar" tasks (pun intended) but yet are very different in certain respects. One main difference is that semantic equivalence and paraphrase are both symmetric relations, i.e., R(u, v) = R(v, u), but relevance and entailment are clearly not. Relevance is distinguished from the others in a few more respects: Queries are usually much shorter than texts (for example, short keyword queries vs. long documents), whereas the two inputs for semantic equivalence, paraphrase, entailment are usually comparable in length (or at the very least, both are sentences). Furthermore, queries can either be short keywords phrases that are rather impoverished in terms of linguistic structure, or well-formed natural language sentences (e.g., in the case of question answering); but for the other three tasks, it is assumed that all inputs are well-formed natural language sentences.
When faced with these myriad tasks, a natural question would be: Do these distinctions matter? With BERT, the answer is, likely not. Abstractly, these are all classification on two input texts 110 (see 107 As a simple example, are apples and oranges similar? Clearly not, because otherwise we wouldn't use the phrase "apples and oranges" colloquially to refer to comparisons between different things. However, from a different perspective, apples and oranges are similar in that they're both fruits. Only point we're trying to make here is that "semantic similarity" is an ill-defined notion that is highly context dependent. 108 Formally, semantic equivalence is conceptualized on an interval scale, so the problem is properly that of regression. However, most models convert the problem into classification (i.e., equivalent or not) and then reinterpret (e.g., renormalize) the estimated probability into the final scale. 109 Note, however, that in practice the reality of paraphrases is much more nuanced. Substitutability needs to be defined in some context, and whether two texts are acceptable paraphrases can be strongly context dependent. Consider a community question answering application: "What are some cheap hotels in New York?" is clearly not a paraphrase of "What are cheap lodging options in London?" A user asking one question would not find the answer to the other acceptable. However, in a slightly different context, "What is there to do in Hawaii?" and "I'm looking for fun activities in Fiji." might be good "paraphrases", especially for a user who was in the beginning stages of planning for a vacation and had not decided on final a destination yet (and hence open to suggestions). As an even more extreme example, "Do I need a visa to travel to India?" and "What immunizations are recommended for travel to India?" would appear to have little to do with each other. However, for a user whose underlying intent was "I'm traveling to India, what preparations are recommended?", answers to both questions are certainly relevant, making them great "paraphrases" in a community question answering application. In summary, there are subtleties that defy simple characterization and are very difficult to model. 110 In the case of Semantic Textual Similarity (STS) tasks, can be converted into classification. Section 3.1) and can be fed into BERT using the standard input template:
where s 1 and s 2 are the two inputs. Provided that BERT is fine-tuned with annotated data that captures the nuance of the desired task, 111 the model should be able to "figure it out" how to model the relevant relationship, be it entailment, paraphrase, or query-document relevance. In fact, there is strong empirical evidence that this is the case, since BERT has been shown to excel at all these tasks.
However, for ranking with dense representations, these task differences may very well be importantand has concrete implications in terms of model design choices. For text ranking, let's consider again what we are trying to estimate:
Does it make sense to use a single η(·) for both the question and the text, given the clear differences between queries and texts (in terms of length, linguistic well-formedness, etc.)? It seems more likely that we should learn separate η q (·) and η d (·) encoders. Specifically, in Figure 19 (a), left, the two "arms" of the network should not share parameters, or perhaps not even share the same architecture. Now, consider reusing much of the same machinery to tackle paraphrase detection, which can be formulated also as an estimation problem:
Here, it would make sense that the same encoder is used for both input sentences, suggesting that models for relevance and paraphrase need to be different? Completely different architectures, or the same network design, but different weights? What about for entailment, where the relationship is not symmetrical? Different researchers have grappled with these issues and provide different solutions, but the key point here is that the exact nature of the task leads to important design choices. It remains an open question whether model-specific adaptations are necessary to capture these differences.
Estimating the relevance of a query to a piece of text is clearly an integral part of the text ranking problem. However, in the context of dense representations, we have found it useful to conceptualize semantic equivalence, paraphrase, and entailment as ranking problems also. In certain contexts, this formulation is natural: in a community question answering application, for example, we wish to find a question from a corpus that is the closest paraphrase to the user's query. Thus, we wish to compute a ranking of questions with respect to the degree of "paraphrase closeness". However, other applications do not appear to fit a ranking formulation: for example, we might wish to determine if two sentences are paraphrases of each other, which certainly doesn't involve ranking. Yet, operationally, these two tasks are exactly the same: we wish to estimate the probability defined in Eq. (51); the only difference is how many candidates how many pairs we perform the estimation over. In other words, under a pointwise classification formulation (which covers everything we discuss in this section), ranking is simply probability estimation over a set of candidates and then sorting by those estimated probabilities. We adopt a ranking conceptualization because it allows us to provide a uniform treatment of these different phenomena.
There is one important implementation detail necessary for ranking with dense representations: solving the nearest neighbor search problem. We specifically draw attention to the problem because this detail is often omitted or glossed over with insufficient detail in many papers today.
Recall that in the setup of the problem we assume the existence of a corpus of texts C = {d i }.
Since a system is provided C "in advance", it is possible to precompute the continuous sparse representation η(d i ) for all texts; slightly abusing notation, we refer to these as η i 's. Although this may be computationally expensive, the task is embarrassingly parallel and can be distributed on an arbitrarily large cluster of machines. The counterpart, η(q), must be computed at query time; also, slightly abusing notation, we refer to this as η q . Thus, the ranking problem is to find the top k most similar η i vectors measured in terms of φ. Similar to search using inverted indexes, this is also a top k retrieval problem. When φ is defined in terms of cosine similarity, inner products, or a handful of other simple metrics, this is known as the nearest neighbor search problem; sometimes also called maximum inner product search if φ is defined as inner products.
The simplest solution to the nearest neighbor search problem is to scan all η i 's vectors and brute force compute φ(η q , η i ). The top k η i 's can be stored in a heap and returned to the user after the scan completes. For small collections, this approach is actually quite reasonable, especially with modern hardware that can take exploit vectorized processing with SIMD instructions. However, this brute force approach becomes impractical for collections beyond a certain point. It might appear that multi-dimensional indexes (e.g., KD-trees) provide a solution to the nearest neighbor search problem, but their standard use case is geospatial applications, and thus they typically do not scale to the size (in the number of dimensions) of the representations that our encoders generate.
Modern scalable solutions to the nearest neighbor search problem are, in fact, approximate. There are a number of ways this can be formalized: for example, Indyk and Motwani [1998] define the k -nearest neighbor search problem as the findings the k closest vectors {η 1 , η 2 , . . . η k } such that the distance of η i to η q is at most (1 + ) times the distance from the ith nearest point to η q . This is typically referred to as the approximate nearest neighbor search problem. The approximation in this context is acceptable in practical applications because φ does not model the end task perfectly to begin with. In search, we are ultimately interested in capturing relevance, and φ is merely a proxy. Thus, an -approximation in distance in 1 − φ usually has no measurable impact on end-user perceptions of quality.
The earliest solutions to approximate nearest neighbor search were based on locality-sensitive hashing [Indyk and Motwani, 1998 , Gionis et al., 1999 , Bawa et al., 2005 , but proximity graph methods methods are generally acknowledged as representing the state of the art today. Approaches based on hierarchical navigable small world (HNSW) graphs Malkov and Yashunin [2020] represent the current state of the art in ANN search based on a popular benchmark. 112 Another popular library for ANN search is Faiss, 113 open-sourced by Facebook [Johnson et al., 2017] .
For the purposes of this survey, we assume the use of a system or library that efficiently solves the approximate nearest neighbor search problem for an arbitrarily large collection of dense vectors, in the same way that we assume the existence of solutions for solving the keyword search problem using inverted indexes at scale (see Section 2.6). There are, of course numerous engineering details to making such functionalities a reality, but they are beyond the scope of this survey.
While the ideas behind word embeddings and continuous representations of words goes back decades, word2vec [Mikolov et al., 2013b,a] is often regarded as first successful implementation that heralded the beginning of neural revolution in natural language processing. Although the paper was primarily about word representations and similarities between words, the authors also attempted to tackle the issue of compositionality and phrase representations. As discussed above, a ranking problem emerges as soon as we try to build dense representations of text beyond individual words and to compare such representations.
With word embeddings, word representations are static vectors (that are fetched from a dictionary) and similarity comparisons are performed (typically) via cosine similarity. However, for any unit of text beyond individual words, there are many options for tackling the representation problem, the composition problem, and the comparison problem. Researchers have grappled with these two core challenges long before transformers were invented, and in fact, many recent advances can be characterized as adaptations of old ideas, but with transformers. Thus, it makes sense to provide a survey of these pre-BERT techniques.
After the initial successes of word embeddings, the next burst of research activity focused on building sentence representations (and in general, representation of longer segments of text); this is precisely the composition problem we mentioned in the introductory remarks in this section. Note that we are concerned with the use case of deriving representations from novel, previously unseen sentences; thus, for example, the paragraph vector representation of Le and Mikolov [2014] is beyond the scope of this discussion since the technique requires training on a corpus to derive representations of paragraphs contained in it. Since natural language has a hierarchical structure, many researchers adopted a hierarchical approach to composing word representations into sentence representations, for example, recursive neural networks [Socher et al., 2013] , and later, Tree-LSTMs [Tai et al., 2015] . As an alternative, Iyyer et al. [2015] proposed Deep Averaging Networks, which disregarded hierarchical structure to compute both sentence-as well as document-level representations based on averaging the embeddings of the individual words and then passing the results through feedforward layers. The authors demonstrated that, for this classification tasks, these simple networks were competitive with, and in some cases, outperformed more sophisticated models while taking far less time to train.
To our knowledge, the first comprehensive evaluation of different aggregation techniques for sentence similarity tasks was the work of Wieting et al. [2016] , who examined six different architectures for generating sentence embeddings, ranging from simple averaging of individual word representations (i.e., mean pooling) to an LSTM-based architecture. The authors examined both an in-domain supervised setting, where models are trained with annotated semantic similarity data drawn from the same distribution as the test data, as well as general purpose, domain independent embeddings for word sequences, using data from a wide range of other domains. While LSTMs performed well with in-domain data, simple averaging vastly outperform LSTMs in out-of-domain scenarios.
Later work examined a range of simple approaches for aggregating individual word representations into representations of larger segments of text: weighted average of word embeddings with learned weights [De Boom et al., 2016] , weighted average of word embeddings followed by modification with SVD [Arora et al., 2017] , random walks [Ethayarajh, 2018] , and different pooling techniques [Shen et al., 2018] . The general conclusion from all these papers seems to be that simple aggregation methods are robust, fast to compute, and effective, either competitive with or outperforming far more complex models.
The references cited above draw mostly from the NLP literature, where researchers were primarily concerned with the problem of estimating semantic similarity. Contemporaneously, IR researchers were exploring similar ideas in the context of document ranking. These were already discussed in Section 1.2.4 in the context representation-based models. For example, the Deep Structure Semantic Model (DSSM) [Huang et al., 2013] constructs vector representations of queries and documents using feedforward networks. For ranking, query and document representations are directly compared using cosine similarity. In fact, the models we describe in Section 4.4 all adopt this basic design, except that the feedback network is replaced by transformers. As another example, the Dual Embedding Space Model (DESM) computes query-document relevance scores by aggregating cosine similarities across all query-document term pairs.
There are a number of other instances of using learned representations for ranking along the lines of DSSM in the literature. Henderson et al. [2017] examined the problem of suggesting email responses in Gmail. Given a training corpus of (message, response) pairs, text encoders using feedforward network are trained to maximize the dot product between the learned representations of the training pairs. Similar ideas for end-to-end retrieval with learned representations are also explored later in Gillick et al. [2018] . With an expansive scope, Wu et al. [2018] proposed StarSpace, with a tagline of "embed all the things", that tries to unify a wide range of tasks (classification, ranking, recommendation, and more) as simple similarity comparisons of learned representations. Zamani et al. [2018] proposed the Standalone Neural Ranking Model (SNRM), which learns sparse query and document representations that can be stored in a normal inverted index for efficient retrieval.
Work on transformer-based dense representation for ranking emerged roughly contemporaneously in 2019 from a few sources. To our knowledge, the earliest paper is Humeau et al. [2019] , dating from April 2019. They introduced the term "bi-encoder" to refer to the simple similarity-based approaches using transformer-encoded representation that we have described above, depicted by Figure 19 (a). In contrast, what they called "cross-encoder" is the standard BERT design that benefits from all-to-all attention across tokens in the inputs, corresponding to Figure 19 (c). That is, a bi-encoder takes two inputs and outputs two representations via η that can be compared with φ, whereas a cross-encoder takes two inputs and outputs a classification decision directly. The focus of their proposed innovation computed candidate embeddings using attention. This idea works for finding the highest scoring sentence in a larger collection. However, polyencoders have the drawback that the score function is not symmetric and the computational overhead is too large for use-cases like clustering, which would require O(n 2 ) score computations. Previous neural sentence embedding methods started the training from a random initialization. In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings. This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods.
SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized sentence embedding. We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEANstrategy), and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN.
In order to fine-tune BERT / RoBERTa, we create siamese and triplet networks (Schroff et al., 2015) to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity.
The network structure depends on the available training data. We experiment with the following structures and objective functions. Classification Objective Function. We concatenate the sentence embeddings u and v with the element-wise difference |u v| and multiply it with the trainable weight W t 2 R 3n⇥k :
where n is the dimension of the sentence embeddings and k the number of labels. We optimize cross-entropy loss. This structure is depicted in Figure 1 .
Regression Objective Function. The cosinesimilarity between the two sentence embeddings u and v is computed (Figure 2 ). We use meansquared-error loss as the objective function.
Triplet Objective Function. Given an anchor sentence a, a positive sentence p, and a negative sentence n, triplet loss tunes the network such that the distance between a and p is smaller than the distance between a and n. Mathematically, we minimize the following loss function: max(||s a s p || ||s a s n || + ✏, 0)
with s x the sentence embedding for a/n/p, || · || a distance metric and margin ✏. Margin ✏ ensures that s p is at least ✏ closer to s a than s n . As metric we use Euclidean distance and we set ✏ = 1 in our experiments.
We train SBERT on the combination of the SNLI (Bowman et al., 2015) and the Multi-Genre NLI Figure 20 : The architecture of Sentence-BERT, from Reimers and Gurevych [2019] . The training architecture for the classification objective is shown on the left. The architecture for inference, to compute similarity scores, is shown on the right. is another model called "poly-encoder", which we detail in Section 4.5. In this survey, we use the bi-encoder vs. cross-encoder terminology introduced by Humeau et al. 114 Following Humeau et al. [2019] , several roughly contemporaneous works closely followed. Lee et al. [2019b] applied the basic bi-encoder design to passage retrieval in the question answering context. Sentence-BERT [Reimers and Gurevych, 2019] applied the bi-encoder design to a number of semantic similarity tasks. applied bi-encoders to aligning cross-lingual entities in the knowledge graph by comparing textual descriptions of those entities. 115 Slightly later, Barkan et al. [2020] investigated how well a BERT-based cross-encoder can be distilled to a BERT-based bi-encoder in the context of sentence similarity tasks.
We provide a more detailed description of Sentence-BERT [Reimers and Gurevych, 2019] as the canonical example of a bi-encoder design for generating semantically meaningful sentence embeddings to be used in large-scale semantic similarity comparisons (see Section 4.1). 116 The overall architecture is shown in Figure 20 , extracted from the authors' paper. The diagram on the left shows how Sentence-BERT is trained: each "arm" of the network corresponds to η(·) in our terminology, which is responsible for deriving a fix-sized embedding for the input (sentences in this case). Reimers and Gurevych [2019] experimented with both BERT and RoBERTa as the "base" model and proposed three options:
• Take the [CLS] vector.
• Mean pooling across all contextual output representations. 114 The bi-encoder design is sometimes referred to as the Siamese architecture or "twin towers"; both terms are potentially problematic in that the former is considered by some to be derogatory and the later evokes negative images of 9/11. The term bi-encoders seem both technically accurate and not associated with negative connotations. 115 The first arXiv submission of poly-encoder unambiguously pre-dates Sentence-BERT, as the latter cites the former. However, Humeau et al.'s original arXiv paper did not appear in a peer-reviewed venue until April 2020, at ICLR [Humeau et al., 2020] . Reimers Humeau et al.'s primary contribution was the poly-encoder architecture, which we discuss in Section 4.5. Furthermore, Humeau et al. evaluated their models on the task of sentence selection in dialogue. While this is no doubt a ranking task, the datasets they used are less popular than the ones examined by Sentence-BERT; thus, results from the latter provide more points of reference for comparison. We selected Sentence-BERT over Lee et al. [2019b] because we felt that Sentence-BERT provided a more detailed treatment of the bi-encoder design. In particular, Reimers and Gurevych provided more detailed ablation analyses, whereas Lee et al. had much of its emphasis elsewhere.
• Max pooling across all contextual output representations.
The first option is obvious, while the other two are clearly applications of previous techniques applied to static embeddings, discussed in Section 4.3. The result is η(Sentence A) = u and η(Sentence B) = v, providing the solution to the representation problem as discussed in Section 4.1.
The two transformer encoders have tied (shared) weights, since the target task is semantic similarity.
Depending on the task, the entire architecture is trained end-to-end as follows:
• For classification tasks, the representation vectors u, v, and their element-wise difference |u − v| are the concatenated and fed to a softmax classifier:
where ⊕ denotes vector concatentation and W t represents the trainable weights; standard cross-entropy loss is used. • For regression tasks, mean squared loss between the ground truth and the cosine similarity between the two sentence embeddings u and v is used. Reimers and Gurevych [2019] additionally proposed a triplet loss structure, which we skip here.
At inference time, the trained encoder η is applied to both sentences, producing sentence vectors u and v. The cosine similarity between these two vectors is directly interpreted as a similarity score; this is shown in Figure 20 , right. That is, in our terminology, φ(u, v) = cos (u, v) . This provides the answer to the comparison problem as discussed in Section 4.1. In a ranking scenario such as semantic search, the representation of all texts in the corpus can be precomputed and indexed using the techniques described in Section 4.2. For a given query, η is first applied to derive a query representation, which is then used as the query vector in nearest neighbor search.
Sentence-BERT was evaluated in three different ways for semantic similarity tasks:
• Untrained. BERT (or RoBERTa) can be directly applied "out of the box" for semantic similarity computation. • Fine-tuned on out-of-domain dataset. Sentence-BERT was fine-tuned on a combination of the SNLI and Multi-Genre NLI datasets [Bowman et al., 2015 , Williams et al., 2018 ]. The trained model was then evaluated on the Semantic Textual Similarity (STS) benchmark [Cer et al., 2017] . • Fine-tuned on in-domain dataset. Sentence-BERT was first fine-tuned on the SNLI and Multi-Genre NLI datasets (per above), then further fine-tuned on the training data of the STS benchmark data before evaluation on the STS benchmark. This is similar to the stage-wise fine-tuning approaches discussed in Section 3.5.5.
Below, we provide a few highlights of the experimental results, but refer readers to the authors' original paper for details:
• Without any training, average pooling of BERT's contextual representations is worse than average pooling of static GloVe embeddings, based on standard metrics for the semantic similarity datasets. Using the [CLS] representation is even worse than average pooling, suggesting that [CLS] alone provides a poor representation of the input "out of the box" (that is, without fine-tuning on task-specific data). • Not surprisingly, out-of-domain fine-tuning helps-and leads to large gains on the STS benchmark over the untrained condition. Note that the first two techniques discussed above can be considered zero-shot or unsupervised, in the sense that the model has never seen labeled data related to the target task. As expected, further in-domain fine-tuning provides another boost in effectiveness, and this result is consistent with the stage-wise fine-tuning approaches discussed in Section 3.5.5. The bi-encoder approach is competitive to the standard cross-encoder design in terms of effectiveness. Sentence-BERT is consistently worse, but in some cases the differences are relatively modest. • Ablation studies showed that, of all the techniques examined, average pooling is the most effective solution to the composition problem in the design of η, slightly better than max pooling or [CLS] . In the untrained setting, the effectiveness of using the [CLS] is quite low, but after fine-tuning, its effectiveness is only slightly behind average pooling.
• For classification tasks, a very interesting finding is the necessity of including |u − v| in the input to the softmax classifier. Recall that when training Sentence-BERT for classification, the input representation comprises the concatenation of the vectors u, v (corresponding to each sentence) and their element-wise difference |u − v|. If the input to the softmax omits |u − v|, effectiveness drops substantially.
Closely related to Sentence-BERT, the contemporaneous work of Barkan et al. [2020] investigated how well a BERT-based cross-encoder can be distilled to a BERT-based bi-encoder in the context of sentence similarity tasks. To do so, the authors train a BERT Large cross-encoder to perform a given task and then distill it into a BERT Large bi-encoder that produces a dense representation of its input by average pooling the outputs of its final four transformer layers. The experimental results support the same general findings from Sentence-BERT: After distillation for a specific task, the bi-encoder student model performs competitively but remains consistently less effective than the cross-encoder. However, as expected, the bi-encoder is significantly more efficient than the teacher cross-encoder.
Following an initial burst of activity on representation learning with transformers in 2019, many researchers became interested in bi-encoder approaches to ranking with dense representations, with a number of papers appearing in rapid succession in 2020. Many of the proposed techniques can be characterized as variations of the basic Sentence-BERT design. We discuss these below:
The dense passage retriever (DPR) of Karpukhin et al. [2020] , reported in April 2020, adopts an architecture quite similar to Sentence-BERT to improve both passage retrieval as well as end-to-end effectiveness in open-domain question answering. The authors adopted a multi-stage architecture that comprises a "retriever" to identify passages from a corpus likely to contain an answer followed by a "reader" that extracts the exact answer spans. In DPR, η takes the [CLS] representation from BERT and φ is defined in terms of inner products. The authors demonstrated that their retrieval approach, followed by a BERT-based extractive question answering component, achieves the state of the art in various popular benchmark datasets. A linear combination of DPR and BM25 scores further boosts retrieval effectiveness. One of the main contributions of Karpukhin et al. is a careful study of how the selection of negative passages during training impacts retrieval effectiveness. They found that the best strategy is to use positive passages from other examples in the same mini-batch together with a passage retrieved by BM25. We do not include results from their paper as the datasets they evaluated on does not overlap with those used by most of the other papers discussed below, and thus there are no common points for comparison.
Contemporaneously with DPR, a number of bi-encoder models for text ranking appeared during the Spring and Summer of 2020, starting with CLEAR [Gao et al., 2020c] in April. Since they have all conducted evaluations on the MS MARCO passage retrieval test collection, meaningful effectiveness comparisons can be made. We present them in chronological order. Gao et al. [2020c] proposed the CLEAR model (short for "Complementing Lexical Retrieval with Semantic Residual Embedding"), which can be described as Sentence-BERT combined with a bag-ofwords retrieval component that is used to both guide training and to produce final relevance scores. To produce a dense representation of a text from the corpus (i.e., the η function), the text is fed to a BERT model; before the usual [CLS] token, another special token, either <QRY> or <DOC>, is prepended to denote the query or document, respectively. The final vector representation is produced by applying average pooling to BERT's output contextual representations. Relevance scores are then computed by taking the weighted sum of the bag-of-words retrieval score, which is produced by BM25, and the inner product between query and document vector representations (i.e., the φ function).
The model is trained using a pairwise hinge loss to maximize the similarity between a given query q and relevant document d + while minimizing the similarity between the query and a non-relevant document d − subject to a minimum margin:
However, rather than using a fixed margin as is common (e.g., m = 1), m is dynamically computed based on the BM25 retrieval scores for the two documents and two parameters c and λ:
The authors stated that this helps specialize the dense representations for capturing semantic similarity, because little loss is incurred when BM25 is able to accurately identify the relevant document. RepBERT, which is short for "representation-focused BERT," is a bi-encoder variant proposed by that can be described as CLEAR without its bag-of-words retrieval component. Like CLEAR, RepBERT produces a dense representation of an input text (i.e., the η function) by processing it with a BERT model and performing average pooling on the contextual representations produced. Unlike CLEAR, RepBERT computes relevance scores (i.e., the φ function) by taking the inner product between query and document representations directly, rather than including a bag-of-words retrieval score. The model is trained using a pairwise hinge loss as in (Eq. 53) with m = 1. did not take advantage of any existing library for nearest neighbor search, and instead compute the similarity between the query and every document.
Building on the bi-encoder design, made the observation that non-relevant documents ranked highly by an exact match ranking method are unlikely to be the same non-relevant documents that are ranked highly by a BERT-based method. This becomes an important issue when BERT-based representations are employed in retrieval with bi-encoders, either as first-stage retrieval or to directly return ranked results. In order to train bi-encoder models more effectively in this context, the authors proposed using approximate nearest neighbor techniques to dynamically identify negative examples that are ranked highly by the bi-encoder model being trained. That is, checkpoints of the bi-encoder model are made at regular intervals during training and used in conjunction with an ANN search to choose "difficult" negative training examples. Their approach, called ANCE for "Approximate Nearest Neighbor Negative Contrastive Estimation," consists of this negative constrastive training approach with a bi-encoder similar to RepBERT. Rather than average pooling, ANCE uses the [CLS] vector as the output of η, the same as Karpukhin et al. [2020] . The model is trained with a negative log likelihood loss rather than the pairwise hinge loss used by RepBERT.
The MS MARCO passage retrieval test collection provides a common comparison point for CLEAR, EPIC, ANCE, and RepBERT, all of which report dev MRR@10 and Recall@1000 on this collection. These results are summarized in Table 27 , copied from the original papers. We additionally include monoBERT baselines from Nogueira and Cho [2019] for context.
We begin by considering MRR@10: ANCE, row (4a), and CLEAR, row (2a), perform similarly, and both substantially outperform RepBERT, row (5), and both EPIC variants, rows (3a) and (3b). When reranking doc2query-T5 in row (3b), EPIC performs similarly to RepBERT in row (5), and performs substantially better than EPIC reranking BM25 in row (3a). None of the approaches can match the effectiveness of monoBERT Base without including a BERT reranker as in row (2d), however, indicating that there is still a gap in effectiveness between bi-encoder and cross-encode designs. Effectiveness in terms of Recall@1k for all these bi-encoder approaches is similar.
MRR@10 Recall@1k (1a) BM25 + monoBERT Large [Nogueira and Cho, 2019] 0.365 -(1b) BM25 + monoBERT Base [Nogueira and Cho, 2019] 0.347 - Finally, there are two other papers worth mentioning that are closely related to work discussed above. The BISON [Shan et al., 2020 ] bi-encoder model ("BM25-weighted Self-Attention Framework") follows a similar approach to RepBERT and ANCE, with a [CLS] vector as the output of η. Rather than being initialized with BERT, BISON consists of a stack of three modified "BISON encoder" layers that are trained directly on Bing query log data. The proposed BISON encoder layer is a transformer encoder variant in which self-attention computations are weighted by term importance, which is calculated using a variant of tf-idf similar to BM25's formula. The model is trained with a standard cross-entropy loss. Unfortunately, BISON was not evaluated on the MS MARCO passage retrieval test collection, and thus a comparison to the techniques in Table 27 is not possible. considered the problem of matching long texts against long texts with a bi-encoder model (e.g., using entire documents as both the query and the unit retrieved). They introduce the MatchBERT approach, which is a SentenceBERT variant similar to RepBERT that compares a query and text using cosine similarity, before introducing the hierarchical SMITH model. SMITH, which is short for "Siamese Multi-depth Transformer-based Hierarchical Encoder," creates sentence-level representations with a stack of two transformer encoder layers; a stack of three transformer encoder layers transforms these sentence representations into a document representation, which is the output of η. Document representations are then compared with cosine similarity. As there are no common points of comparison between this work an the others discussed above, we do not present results from MatchBERT or SMITH.
In the bi-encoder approaches discussed above, the representations u and v derived from the encoder η are compared using a simple similarity metric like cosine similarity, that is, φ(u, v) = cos(u, v). For ranking, since the representation of the candidate texts can be precomputed, approximate nearest neighbor search techniques (Section 4.2) can compute the k nearest vectors to the query vector in terms of φ; this is exactly the top k ranking in the context of dense representations. While usually much faster, bi-encoder approaches are invariably less effective than cross-encoder approaches (e.g., monoBERT) because cross-encoders can exploit relevance signals derived from attention between the query and candidate text at each transformer encoder layer. Thus, the tradeoff with bi-encoders is invariably sacrificing (usually quite a bit of) effectiveness for (large) efficiency gains. The Poly-encoder combines the strengths of the Bi-encoder and Cross-encoder by both allowing for caching of candidate representations and adding a final attention mechanism between global features of the input and a given candidate to give richer interactions before computing a final score.
others are negatives taken from the training set. Unlike in the Bi-encoder, we cannot recycle the other labels of the batch as negatives, so we use external negatives provided in the training set. The Cross-encoder uses much more memory than the Bi-encoder, resulting in a much smaller batch size.
Inference speed Unfortunately, the Cross-encoder does not allow for precomputation of the candidate embeddings. At inference time, every candidate must be concatenated with the input context and must go through a forward pass of the entire model. Thus, this method cannot scale to a large amount of candidates. We discuss this bottleneck further in Section 5.4.
The Poly-encoder architecture aims to get the best of both worlds from the Bi-and Cross-encoder. A given candidate label is represented by one vector as in the Bi-encoder, which allows for caching candidates for fast inference time, while the input context is jointly encoded with the candidate, as in the Cross-encoder, allowing the extraction of more information.
The Poly-encoder uses two separate transformers for the context and label like a Bi-encoder, and the candidate is encoded into a single vector y cand i . As such, the Poly-encoder method can be implemented using a precomputed cache of encoded responses. However, the input context, which is typically much longer than a candidate, is represented with m vectors (y 1 ctxt ..y m ctxt ) instead of just one as in the Bi-encoder, where m will influence the inference speed. To obtain these m global features that represent the input, we learn m context codes (c 1 , ..., c m ), where c i extracts representation y i ctxt by attending over all the outputs of the previous layer. That is, we obtain y i ctxt using: y i ctxt = X j w c i j h j where (w c i 1 , .., w c i N ) = softmax(c i · h 1 , .., c i · h N ) 5 Figure 21 : Architecture of the poly-encoder, taken from [Humeau et al., 2020] .
One main source of the large effectiveness differences between bi-encoders and cross-encoders is the use of simple similarity metrics as φ. The natural follow-up question is whether we can make φ a bit richer to better capture the complexities of relevance. The design of φ, however, is constrained by current nearest neighbor search techniques if we wish to take advantage of off-the-shelf libraries to perform ranking directly, or alternatively, we would need to build custom nearest neighbor search capabilities from scratch. Therein lies the challenge.
The PreTTR (Precomputing Transformer Term Representations) model [MacAvaney et al., 2020b] illustrates a hybrid design between a bi-encoder and a cross-encoder. Starting with monoBERT, the authors modified the all-to-all attention patterns of BERT to eliminate attention between the query and the candidate text. That is, terms in the candidate text cannot attend to terms in the query, and vice versa; this is accomplished by a mask. If this mask is applied to all the layers in BERT, we have essentially "cleaved" monoBERT into disconnected networks for the query and the candidate text. In this case, the representation for the candidate texts (i.e., all texts from the corpus) can be precomputed, and the overall design is quite close to a bi-encoder. However, the attention mask can be applied to only some of the transformer encoder layers. Suppose we apply it to all but the final layer: this means that the representation of the candidate text just before the final transformer encoder layer can be precomputed. At inference time, the model can look up the precomputed representation and only needs to apply inference with the final layer; inference on the query, however, needs to proceed through all the layers. Since the candidate texts are usually much longer than the queries, this yields large savings in inference latency. By controlling the number of layers the attention mask is applied to, it is possible to trade off effectiveness for efficiency.
To summarize, in PreTTR, the choice of φ is the "upper layers" of a monoBERT model, while η d for the texts in the corpus come from the "lower layers" of a monoBERT model (via attention masking). Contemporaneously, Gao et al. [2020a] had similar intuitions as well. While these represent nice illustrations of a hybrid model that lies between bi-encoders and cross-encoders, the design remains rooted in a reranking setup, with candidate texts coming from some first-stage retrieval technique (presumably based on keyword search). Below, we describe two approaches that develop more complex designs for the comparison function φ to increase the effectiveness of ranking using dense transformer-based representations.
Humeau et al. [2020] was, to our knowledge, the first to have proposed successful neural architectures for ranking using transformer-based dense representations, pre-dating Sentence-BERT [Reimers and Gurevych, 2019] (Section 4.4); cf. [Humeau et al., 2019] . They introduced the bi-encoder and cross-encoder terminology that we have adopted in this survey, as baselines for their proposed innovation, called the poly-encoder model, shown in Figure 21 .
The poly-encoder model aimed to improve the effectiveness of bi-encoders at the cost of a (relatively small) decrease in efficiency, using a comparison φ function that takes advantage of multiple representations of the candidate text. Confusingly, Humeau et al. [2020] called their query the "candidate" and a candidate text the "context"; for consistency, we have adopted their terminology. In bi-encoders, η reduces an input text to single fixed-width vector: in Sentence-BERT, this is accomplished by mean pooling the contextual representations from BERT. In contrast, poly-encoders derive m vector representations from the input context, (y 1 ctx , . . . , y m ctx ), where m is a hyperparameter that controls the tradeoff between inference speed and effectiveness. This is accomplished by learning m context codes (c 1 , . . . , c m ), where c i extracts the representation y i ctx by attending over all outputs of the previous layer. In more detail:
The m context codes are learned from random initialization. Thus, poly-encoder generates m representations of the context (candidate text) that "view" the input in different ways.
In contrast, only one representation y cand is generated from the candidate (query). Like Sentence-BERT, Humeau et al. [2020] tried both taking the contextual representation of the [CLS] token as well as average pooling the contextual representation of the entire sequence, and in addition, experimented with average pooling over only the first t contextual representations. Ablation studies showed that [CLS] gave slightly better results. The m representations of the contexts (y 1 ctx , . . . , y m ctx ) are aggregated into a single context vector y ctx by attending over them with y cand . That is:
where (w 1 , . . . , w m ) = softmax(y cand · y 1 ctx , . . . , y cand · y m ctx )
The final relevance scores is then y ctx · y cand . The entire model is trained end-to-end with task-specific labeled data; we refer readers to the authors' original paper for details.
It is worth noting that Humeau et al. [2020] did not discuss whether poly-encoders are compatible with approximate nearest neighbor search methods like those discussed in Section 4.2. Their evaluation of efficiency only included reports of inference latency over fixed sets of candidates from the training examples, and did not include the end-to-end task of ranking texts from a much larger corpus. 117 Because the computation of y ctx requires attending over the query representation, direct application of ANN search techniques does not seem possible. We can, however, envision reasonable ways around this issue, by separately indexing the individual "views" (y 1 ctx , . . . , y m ctx ) of texts in the corpus, using ANN search to generate a list of candidates based on these, and computing y ctx only from these candidates; note that this solution brings us back to a multi-stage architecture. To our knowledge, this design (as well as other possible solutions) have not been evaluated, and so their tradeoff between effectiveness and efficiency is unknown. Humeau et al. [2020] compared poly-encoders with bi-encoders and cross-encoders in the context of response selection, which is the task of retrieving appropriate responses to an utterance in a conversation [Lowe et al., 2015 , Yoshino et al., 2019 , Dinan et al., 2019 . That is, conversational utterances serve as queries and the model's task is to identify which piece of text to "say next." While this task differs from ad hoc retrieval, it is nevertheless a ranking task. We omit duplicating results from their paper here since none of the other methods discussed in this survey use those datasets, and thus there are no other points of reference to provide context. However, the authors empirically validated that poly-encoder models were more effective than bi-encoders and more efficient than cross-encoders (at least in terms of inference latency; see discussion above). We are not aware of work applying poly-encoders to ad hoc retrieval tasks, particularly the end-to-end setting of ranking texts from a large corpus, but this is a promising direction worth additional exploration.
While poly-encoders to our knowledge was the first to develop the intuition of exploiting multiple representations, this approach has been pursued by others as well. For example, Luan et al. [2020] proposed building m representations of each text in the corpus in a manner that is independent of the query. The query-document score is simply the largest inner product between the query and any of these m representations. Unlike with poly-encoders, the encoder η does not depend on the query, so this approach to ranking can directly take advantage of ANN search libraries, and the authors reported scores on the MS MARCO passage retrieval test collection that are comparable to the best scores reported in Table 27 .
For a range of ad hoc retrieval tasks, it is clear that transformer-based models in a multi-stage reranking architecture yield highly effective results. The major downside, as already discussed in Section 3, is high query latency due to slow and computationally expensive transformer inference. Based on the analysis of , monoBERT is about two orders of magnitude slower than pre-BERT neural ranking models (see Section 3.6). A high-level summary of current approaches to accelerating transformer-based inference is that in order to achieve query latencies comparable to pre-BERT neural rankers, large sacrifices in effectiveness must be made-for example, as in the TK/CK family of approaches (see Section 3.6.2).
In other words, there is large gap in the effectiveness/efficiency tradeoff space between, for example, monoBERT and the TK model. Because representations of texts in the corpus can be precomputed, ranking techniques with dense vector representations can potentially offer other compelling operating points in the tradeoff space, perhaps bridging this gap. However, we have seen that simple implementations of the comparison function φ (e.g., cosine similarity) in a bi-encoder design still sacrifices a large amount of effectiveness compared to multi-stage ranking architectures. Poly-encoders tried to implement a more sophisticated φ, but has a number of limitations, as we have already noted.
Nevertheless, this general approach of ranking with more sophisticated comparison functions is promising and has attracted much interest from researchers. The next breakthrough along these lines was the work of Khattab and Zaharia [2020] , who were the first to demonstrate that ranking methods based on dense representations can achieve levels of effectiveness that are competitive to a cross-encoder design, but at a fraction of the query latency, narrowing the gap with pre-BERT neural models. Their method, dubbed ColBERT, achieves a very compelling tradeoff between effectiveness and efficiency, filling exactly the gap referenced above.
Khattab and Zaharia began by noting that previous methods represented each text from the corpus using a single dense vector. Like poly-encoders, the inspiration was to generate more fine-grained representations for queries and texts from the corpus. ColBERT computes a dense vector for each token from the query or the text in the corpus based on contextual representations from BERT.
More formally, given a text t consisting of a sequence of tokens [t 1 , ..., t n ], ColBERT computes a matrix η([t 1 , ..., t n ]) ∈ R n×D , where n is the number of tokens in the text and D is dimension of each token representation. In other words, the output of the η encoder is a matrix, not just a vector. Interestingly, the model uses the same BERT model to encode queries and texts from the corpus. To distinguish them, however, a special token [Q] is prepended to queries and a separate token [D] to texts from the corpus. As with other ranking approaches based on dense vectors, representations of texts in the corpus can be computed offline since they do not depend on the query.
In order to control the vector dimension D, a linear layer without activation is added on top of the last layer of the BERT encoder. This allows representations of the texts from the corpus to be stored using less space and to be transferred to CPU or GPU memory more quickly for similarity comparisons (more discussion on this later). Additionally, the vector representation of each token is normalized to an unitary L2 norm; this makes vector dot products equivalent to cosine similarities.
At inference time, a query q with terms [q 1 , ..., q m ] is converted to η([q 1 , ..., q m ]) ∈ R m×D . A similarity (relevance) score s q,d is computed for each text d from the corpus as follows:
where η(t) i is the vector representing the i-th token of the text t (either the query or a text from the corpus). Since each of these vectors has unit length, the similarity is the sum of maximum cosine similarities between each query term and the "best" matching term from the text in the corpus (the authors called this the "MaxSim" operation). The basic scoring function described above assumes that relevance scores are computed over all texts in the corpus; retrieving the top k can accomplished by sorting the results in decreasing order according to s q,d . To directly perform top k ranking against all texts in a large corpus when a brute-force computation of the similarity values s q,d , ∀d ∈ C is not practical, ColBERT adopts a more efficient two-stage retrieval method. Prior to retrieval, the representation of each term in the corpus is indexed using Facebook's Faiss ANN search library, 118 where each vector retains a pointer back to its source (i.e., the text in the corpus that contains it). At query time, ranking proceeds as follows: In the first stage, each query term embedding η(q) i is issued concurrently as a query and the top k texts from the corpus are retrieved (e.g., k = k/2), by following the pointer of each term vector back to its source. The total number of candidate texts is thus m × k , with K ≤ m × k of those being unique. The intuition is that these K documents will likely be relevant to the query because their embeddings are highly similar to at least one of the query embeddings. In the second stage, these K unique documents will be scored using all query embeddings using Eq (57). As an additional optimization, ColBERT takes advantage of a cluster-based feature inside Faiss to increase the efficiency of the vector searches.
Somewhat ironic here is that in order for ColBERT to scale real-world corpora, a multi-stage architecture is required, which breaks the elegance of a single-stage ranking with ANN search based on bi-encoders. In effect, the authors have replaced first-stage retrieval with an inverted index with first-stage search using an ANN search library.
The ColBERT model is trained end-to-end using the following loss: loss = − log e s q,d + e s q,d + + e s q,d − ,
where d + and d − are relevant and non-relevant documents to the query q, respectively.
An important additional trick used by ColBERT is to append [MASK] tokens to queries that are shorter then a predefined length. The authors argued that this provides a form of query augmentation, since these extra tokens allow the model to learn to expand queries with new terms or to reweight existing terms based on their importance for matching the query. Khattab and Zaharia [2020] evaluated ColBERT on the MS MARCO passage retrieval test collection (among other datasets), which supports a fair comparison with other techniques presented in this survey. Results summarized from their paper are presented in Table 28 . Latency measurements were performed over an NVIDIA 32GB V100 GPU. Row (1a) and (1b) report the standard BM25 baseline and with monoBERT Large reranking, respectively. Row (2) copies the author's report of FastText + ConvKNRM, which can be characterized as a competitive pre-BERT neural ranking model. Row (3) reports the result of doc2query-T5. Row (4) reports effectiveness and query latency figures for ColBERT. We see that ColBERT approaches the effectiveness of monoBERT Large , row (1b) in terms of MRR@10 but is approximately 70× faster on a modern GPU. While ColBERT is more effective than either doc2query-T5 and ConvKNRM, it is still 5× slower. Note that ConvKNRM is evaluated on a GPU, whereas doc2query-T5 runs on a CPU.
As a summary, results show that, in terms of query latency, ColBERT has indeed closed much of the gap between monoBERT and pre-BERT neural ranking models. It is able to accomplish this feat with only modest degradation in effectiveness, which is impressive. However, it is still multiple times slower than pre-BERT neural models and doc2query, although ColBERT is much more effective than both. These results show that ColBERT populates a compelling point in the effectiveness/efficiency tradeoff space. Note however, that in terms of multi-stage architectures, monoBERT Large is but a baseline. There exist even more effective methods, for example, duoBERT (Section 3.4.1, and the top leaderboard entries in the MS MARCO passage retrieval task now report MRR@10 above 0.400. Thus, ranking with dense vector representations still have a ways to catch up.
Furthermore, there is one significant drawback of ColBERT: the space needed to store document representations. For example, the MS MARCO passage corpus contains 8.8M passages. Using nice round numbers, suppose that each passages has on average 200 tokens, each token is represented by a 128-dimensional vector, and we use 4 bytes to encode each dimension. We then need 8.8M passages × 200 tokens × 128 dim × 4 bytes = 900 GB of space! In practice, this amount can be reduced by using fewer bits to represent each dimension and by compressing vectors with a lossless compression algorithm. In fact, Khattab and Zaharia report that "only" 156 GB is required to store the corpus, likely due to some of these optimizations. Nevertheless, this is still almost two orders of magnitude larger the 2.5 GB required by the index of the same collection by Lucene.
This analysis raise an interesting point that we have so far not considered: the space aspect of efficiency. Thus, it seems that more accurate characterization of the tradeoff space must take into account effectiveness of the results, query latency, computational cost at the indexing phase, and storage cost.
It is quite remarkable that BERT debuted in October 2018, only around two years ago. Taking a step back and reflecting, the field has seen an incredible amount of progress in a short amount of time. As we have noted in the introduction and demonstrated throughout this survey, the foundations of how to apply BERT and other transformer architectures to ranking are already quite sturdythe improvements in effectiveness attributable to, for example, the simple monoBERT design, are substantial, robust, and have been widely reproduced in many tasks. We can confidently assert that the state of the art has significantly advanced over this time span , which has been notable in the amount of interest, attention, and activity that transformer architectures have generated. These are exciting times!
We are nearing the end of this survey, but we are still far from the end of the road in this line of research-there are still many open question, unexplored directions, and much more work to be done. The remaining pages below represent our attempt to prognosticate on what we see in the distance, but we begin with some remarks on material we didn't get a chance to cover.
Despite the wealth of obvious connections between transformer-based text ranking models and other NLP tasks and beyond, there are a number of notable content omissions in this survey. As already mentioned at the outset in Section 1.3, we intentionally neglected coverage of other aspects of information access such as question answering, summarization, and recommendation.
The omission of question answering, in particular, might seem particularly glaring, since at a high level the difference between document retrieval, passage retrieval, and question answering can be viewed as granularity differences in the desired information. Here we draw the line between span extraction and ranking explicitly defined segments of text. Standard formulations of question answering (more precisely, factoid question answering) require systems to identify the precise span of the answer (for example, a named entity or a short phrase) within a larger segment of text. These spans are not pre-defined, in the same way that the unit of ranking is defined explicitly by the corpus.
Given this perspective, we have intentionally neglected coverage of any work in question answering focused on span extraction. This decision is consistent with the breakdown of the problem in the literature. For example, Chen et al. [2017a] outlined a "retriever-reader" framework: The "retriever" is responsible for retrieving candidates from a corpus that are likely to contain the answer and the "reader" is responsible for identifying the answer span. This is just an instance of the multi-stage ranking architectures we have discussed in depth; one can simply imagine adding a reader to any existing multi-stage design to convert a search system into a question answering system. The design of retrievers squarely lies within the scope of this survey, and indeed we have interwoven instances of such work in our narrative, e.g., DPR [Karpukhin et al., 2020] in Section 4.4.2 and Cascade Transformers [Soldaini and Moschitti, 2020] in Section 3.4.2.
Nevertheless, the impact of BERT and other transformer architectures on span extraction in question answering (i.e., the "reader") has been at least as significant as the impact of transformers in text ranking. Paralleling Nogueira and Cho [2019] , BERTserini [Yang et al., 2019c] was the first instance of applying a BERT-based reader to the output of a BM25-based retriever to perform question answering directly on Wikipedia. Prior to this work, BERT had been applied in a reading comprehension setup where the task is to identify the answer in a given document (i.e., there was no retriever component), e.g., . A proper treatment of the literature here would take up another volume, 119 but see for a tutorial on recent developments.
Another closely related emerging thread of work that we have not covered lies at the intersection of question answering and document summarization. Like search and question answering, summarization techniques have been heavily driven by transformer architecture in recent years, particularly sequence-to-sequence models given their natural fit (i.e., full-length document goes in, summary comes out Zhang et al. [2020c] . In the query-focused summarization variant of the task [Dang, 2005] , target summaries are designed specifically to address a user's information need. Techniques based on passage retrieval can be viewed as a (strong) baseline for this task, e.g., select the most relevant 119 Perhaps the topic for our next survey? sentence(s) from the input text(s). Although most recent work on question answering is extractive in nature (i.e., identifying a specific answer span in a particular piece of text), researchers have begun to explore abstractive question answering, where systems may synthesize an answer that is not directly contained in any source document [Izacard and Grave, 2020] . Abstractive approaches have the potential advantage in providing opportunities for the underlying model to synthesize evidence from multiple sources. At this point, the distinction between query-focused summarization, passage retrieval, and abstractive question answering become quite muddled-but in a good way, because they present an exciting melting pot of closely related ideas, from which interesting future work is bound to emerge.
While we regret that this survey does not present a more comprehensive look at the broad panoply of information access technologies, especially those exciting developments described above, ultimately this reduction in scope was necessary. Otherwise, our work would never have been completed in a timely manner.
Looking into the future, we are able to identify a number of open research questions, which we discuss below. These correspond to threads of research that are being actively pursued right now, and given the rapid pace of progress in the field, we would not be surprised if there are breakthroughs in answering these question by the time a reader consumes this survey.
Transformers for ranking: apply, adapt, or redesign?
At a very high level, text ranking with transformers can be tackled by three distinct approaches:
1. apply existing transformer models with minimal modifications-exemplified by monoBERT and ranking with T5;
2. adapt existing transformer models, perhaps adding additional architectural elementsexemplified by CEDR and Parade; or, 3. redesign transformer-based architectures from scratch-exemplified by the TK/CK models.
Which is the "best" approach? And to what end? Are we seeking the most effective model, without any considerations regarding efficiency? Or alternatively, are we searching for some operating point that balances effectiveness and efficiency?
There are interesting and promising paths forward for all three approaches: The first approach ("apply") allows researchers to take advantage of innovations in text processing more broadly essentially "for free", and fits nicely with the "more data, larger models" strategy; see additional discussions below. The last approach ("redesign"), on the other hand, requires researchers to reconsider each future innovation specifically in the context of text ranking and assess its applicability. However, this approach has the advantage in potentially stripping away all elements unnecessary for the problem at hand, thereby achieving possibly better effectiveness/efficiency tradeoffs (for example, the TK/CK models). The second approach ("adapt") tries to chart a course in the middle ground, retaining a "core" that can be swapped for a better model that comes along later (for example, PARADE swapping out BERT for ELECTRA).
In the design of transformer models for ranking, it is interesting to see the evolution of techniques than resembles the back-and-forth swing of a pendulum. Pre-BERT neural ranking models were characterized by a diversity of designs, utilizing a wide range of convolutional and recurrent components. In the move from pre-BERT interaction-based ranking models to monoBERT, all these architectural components became subsumed in the all-to-all attention mechanisms in BERT. For example, convolutional filters with different widths and strides didn't appear to necessary anymore, replaced in monoBERT the architecturally homogeneous transformer layers. However, we are now witnessing the reintroduction of specialized components to explicitly capture intuitions important for ranking-for example, the hierarchical design of PARADE (Section 3.3.4) and the reintroduction of similarity matrices in TK/CK (Section 3.6.2).
These points apply equally to ranking with dense representations. Current models either "apply" off-the-shelf transformers with minimal manipulation of their output (e.g., mean pooling in Sentence-BERT) or "adapt" the output of off-the-self transformers with other architectural components (e.g., poly-encoders). In principle, it would be possible to completely "redesign" transformer architectures for ranking on dense representations, similar to the motivation of TK/CK for reranking. However, we are not aware of any such efforts to date.
So, does the future lie with apply, adapt, or redesign? All three approaches are promising, and we see the community continuing to pursue all three paths in the future.
What's the future of multi-stage ranking architectures? What's the role of dense vector and sparse vector techniques?
While the organization of this survey might suggest that these are independent questions, we believe that in the future these questions will become increasingly intertwined. It is true that one motivation for ranking with dense representations (Section 4) is to replace an entire multi-stage ranking pipeline with a single retrieval stage based on approximate nearest neighbor search. It is also the case that multi-stage ranking designs today start with candidate generation based on sparse vector representations (e.g., bag-of-words retrieval using inverted indexes), followed by reranking with dense vector representations (e.g., using a cross-encoder design). However, it is certainly possible to mix and match these architectural patterns.
Dense vector ranking techniques usually trade off effectiveness for efficiency-the ranking quality isn't quite as good as a full cross-encoder design in a multi-stage architecture, but the ranking latency is much lower. In truth, though, for a comparison function φ more complex than cosine similarity, inner products, or a handful of other metrics, ranking is already multi-stage. ColBERT (Section 4.5.2) in the end-to-end setting, for example, uses an ANN library to first gather candidates that are then reranked. Furthermore, with any design based on cosine similarity, we can further improve ranking effectiveness by reranking its output with a cross-encoder. The query latency in such a design may be acceptable because the dense vector retrieval techniques have already reduced the candidate set quite substantially. In this case, we're back to a multi-stage ranking architecture, and in fact this is the very motivation of multi-stage designs to begin with (see Section 3.4).
From the other perspective, we could just as easily start with a multi-stage ranking architecture and replace candidate generation using inverted indexes with candidate generation using approximate nearest neighbor search, directly incorporating dense vector representations. In fact, attempts to do away with retrieval based on sparse vector representations date back to latent semantic analysis from the 1990s [Deerwester et al., 1990] . A more recent example is the work of Nakamura et al. [2019] : in a standard design where BM25-based first-stage retrieval feeds DRMM for reranking, the authors experimented with replacing first-stage retrieval with approximate nearest-neighbor search based on representations from a deep averaging network [Iyyer et al., 2015] . Unfortunately, the end-to-end effectiveness was much worse, but this was "pre-BERT", prior to the advent of the latest transformer techniques. More recently, had more success replacing candidate generation using BM25 with candidate generation using dense vectors derived from the transformer-based Universal Sentence Encoder (USE) [Cer et al., 2018] . They demonstrated that a multi-stage architecture with a first-stage ANN can offer better tradeoffs between effectiveness and efficiency for certain tasks (vs. a standard inverted index), particularly those involving shorter segments of text.
We believe that there will always be multi-stage ranking architectures, for the simple reason that they can incorporate any innovation that adopts a single-stage approach (using dense vector representations or otherwise) as a stage, and then try to improve upon its results with further reranking. However, most multi-stage approaches currently mix sparse and dense representations, and in any such design, representation mismatches will always remain an issue (see discussion in Section 4.1). That is, the types of texts that a model is trained with (in isolation) may be very different from the types of text it sees when inserted into a multi-stage architecture. 120 We raised this issue in Section 3.2, although the mismatch between BM25-based first-stage retrieval and BERT's contextual representation doesn't seem to have negatively impacted effectiveness. Nevertheless, provided an example of an explicit attempt to address these issues. Unless we can design multi-stage models that are end-to-end differentiable, piecing together different stages will remain rather ad hoc, requiring empirical validation. While there is previous work that examines how multi-stage ranking pipelines can be learned [Wang et al., 2010 , Xu et al., 2012 , to our knowledge this is an important issue that remains unexplored in the context of transformer architectures.
What are the evolving roles of zero-shot learning, transfer learning, domain adaptation, and taskspecific fine-tuning?
Ultimately, all solutions are domain and task specific: a domain/task combination can be as broad as general web queries on snippets of web pages (as captured by the MS MARCO passage retrieval test collection) or as as narrow as information needs of domain experts on scientific articles (as captured by the TREC-COVID challenge).
At the outset, BERT's prescription was to start with a pretrained model and then fine tune on domainand task-specific data. However, researchers have explored many alternate recipes, both for language analysis tasks in general and text ranking in particular. Variations include: additional self-supervised pretraining on the target corpora; fine-tuning in different orders on out-of-task or out-of-domain annotated data; fine-tuning on synthetic data generated via weak supervision signals; and, fine-tuning on subsets of a larger dataset. Of course, all these techniques can be sequenced in different orders, yielding a combinatorial explosion in the space of design choices that a researcher faces.
Yet, there is another approach, which is to simply build bigger models. The GPT family [Brown et al., 2020] continues to push the frontiers of larger models, more compute, and more data. Just to be clear, while this approach has a number of obvious problems that are beyond the scope of this discussion, it nevertheless demonstrates impressive effectiveness on a variety of natural language tasks, both in a zero-shot setting and prompted with only a few examples. For text ranking specifically, the T5-3B experiments by Nogueira et al. [2020] (where 3B refers to 3 billion parameters) pushed the limits of model size and demonstrated effective zero-shot relevance classification on specialized information needs related to COVID-19 (on scientific articles) when trained only on MS MARCO passage data (general web domain) .
What do we do when faced with this dizzying variety of choices? To be honest, we don't know, and there are to date no clear answers that consider effectiveness, robustness, the amount of training data required, and other considerations. Sorting through these myriad possibilities is an important research direction.
How can transformers help break the language barrier in information access?
It goes without saying that the web is multilingual and that speakers of all languages have information needs that would benefit from information access technologies. Yet, the techniques discussed in this survey have focused on English. We can, and we should, broaden the scope of exploration; not only would these studies be technically interesting, but potentially impactful in improving the lives of users around the world.
Attempts to break the language barrier in information access can be divided into two related efforts: mono-lingual retrieval in non-English languages and cross-lingual retrieval. In the first case, we would like to support non-English speakers searching in their own language-for example, Urdu queries retrieving from Urdu documents. Of course, Urdu ranking models can be built if there are sufficient resources (test collections) in Urdu, as many supervised machine learning techniques are language agnostic, but as we have already discussed (see Section 2.1), building test collections is an expensive endeavor and thus not a cost-effective solution if we wish to support the some six thousand languages that are spoken in the world today. Can we leverage relevance judgments and data that are available in high-resource languages (English, for example) to benefit languages for which we lack sufficient resources?
The second information access scenario is cross-lingual retrieval, where the language of the query and the language of the documents differs. Such technology, especially coupled with robust machine translation, can unlock stores of knowledge for users that they don't otherwise have access to. For example, Bengali speakers in India can search for information in English web pages, and a machine translation system can then translate the pages into Bengali for the users to consume. Even with imperfect translations, it is still possible to convey the gist of the English content, which is obviously better than nothing if the desired information doesn't exist in Bengali. Note that cross-lingual retrieval techniques can also benefit speakers of English and other high-resource languages: for example, in Wikipedia, it is sometimes the case that "localized versions" of articles contain more information than the English versions. The Hungarian language article about a not-very-well-known Hungarian poet or a location in Hungary might contain more information than the English versions of the articles. In this case, English speakers would benefit from cross-lingual retrieval techniques searching in Hungarian.
Explorations of multilingual applications of BERT for information access have already begun. Google's October 2019 blog post 121 announcing the deployment of BERT (which we referenced in the introduction) offered some tantalizing clues:
We're also applying BERT to make Search better for people across the world. A powerful characteristic of these systems is that they can take learnings from one language and apply them to others. So we can take models that learn from improvements in English (a language where the vast majority of web content exists) and apply them to other languages. This helps us better return relevant results in the many languages that Search is offered in. For featured snippets, we're using a BERT model to improve featured snippets in the two dozen countries where this feature is available, and seeing significant improvements in languages like Korean, Hindi and Portuguese.
Regarding the first point, what Google was referring to 122 may be something along the lines of what Shi and Lin [2019] demonstrated in November 2019. They presented experimental results using an extension of Birch (see Section 3.3.1) showing that multilingual BERT is able to transfer models of relevance across languages. Specifically, it is possible to train BERT ranking models with English data to improve ranking quality in (non-English) monolingual retrieval as well as cross-lingual retrieval, without any special processing. In contemporaneous work, these findings were confirmed by MacAvaney et al. [2020e] . The second point in Google's blog post likely refers to multi-lingual question answering, which recently has benefited from the introduction of new datasets [Cui et al., 2019 , Liu et al., 2019a .
Although some of the early neural ranking approaches did explore cross-lingual retrieval [Vulić and Moens, 2015] and new research on this topic continues to emerge in the neural context [Yu and Allan, 2020] , we have not found enough references in the context of transformers to provide a detailed treatment in a dedicated section. However, moving forward, this is a fertile ground for exploration.
What can text retrieval and text ranking bring to transformers?
This survey is mostly about applications of transformers to text ranking. That is, how can pretrained models be adapted in service of information access tasks. However, there is an emerging thread of work, best represented by REALM [Guu et al., 2020] , that seeks to integrate text retrieval and text ranking directly into model pretraining. The idea is based on the observation that BERT and other pretrained models capture a surprisingly large number of facts, simply as a side effect of the masked language model objective. Is it possible to better control this process so that facts are captured in a more modular and interpretable way? The insight of REALM is that prior to making a prediction about a masked token, the model can retrieve and attend over related documents from a large corpus such as Wikipedia. Retrieval is performed based on dense representations like those discussed in Section 4. Similar intuitions have also been explored by others. For example, Wang and McAllester [2020] viewed information retrieval techniques as a form of episodic memory for augmenting GPT-2. In the proposal of Wu et al. [2020a] , a "note dictionary" that saves the context of a rare word in maintained during pretraining, such that when the rare word is encountered again, the saved information can be leveraged.
Thus the question is not only "What can transformers do for text ranking?" but also "What can text ranking do for transformers?" No doubt, the answers will be exciting.
We have seen again and again throughout this survey that much recent work seems to be primarily adaptation of old ideas, many of which are decades old. For example, monoBERT, which heralded the BERT revolution for text ranking, is just pointwise relevance classification-dating back to the late 1980s [Fuhr, 1989 ]-but with more powerful models.
To be clear, we don't think there is anything "wrong" (or immoral, or unethical, etc.) with recycling old ideas: in fact, the filmmaker Kirby Ferguson, claims that "everything is a remix". He primarily refers to creative endeavours such as music, but the observation applies to science and technology as well. Riffing off Picasso's quote "Good artists copy, great artists steal", Steve Jobs once said, "We have always been shameless about stealing great ideas". 123 The concern, however, is when we as scientists lose touch with previous work and the rich body of literature that came before, in the case of natural language processing, often for the simple reason that they don't use deep learning.
In "water cooler conversations" around the world and discussions on social media, (more senior) researchers who were trained before the advent of deep learning often complain, and only partly tongue-in-cheek, that most students today don't believe that natural language processing existed before neural networks. It is not uncommon to find deep learning papers today that cite nothing but other deep learning papers, and nothing before the early 2010s. Isaac Newton is famous for saying "If I have seen further than others, it is by standing upon the shoulders of giants." We shouldn't forget whose shoulders we're standing on, but unfortunately, often we are. 124
On a practical note, this means that there are likely still plenty of gems in the literature hidden in plain sight; that is, old ideas that everyone has forgotten, that has acquired new relevance in the modern context. It is likely that many future innovations will be remixes!
At last, we have come to the end of our survey. The information access problem has plagued civilizations since shortly after the invention of writing, when humankind's collective knowledge outgrew the memory of its elders. Although the technologies have evolved over the millennia, from clay tablets to scrolls to books, and now electronic information stored digitally, the underlying goals have changed little: we desire to develop tools, techniques, and processes to address users' information needs. The academic locus of this quest with computers, which resides in the information retrieval community (and to a lesser extent, the natural language processing community), has only been around for roughly three quarters of a century-a baby in comparison to other academic disciplines.
We can trace the evolution of information retrieval through major phases of development (exact match, learning to rank, pre-BERT neural networks), as described in the introduction. No doubt we are currently in the age of BERT and transformers. Surely, there will emerge new technologies that completely supplant these models, bringing in the dawn of a new age. Nevertheless, while we wait for the next revolution to happen, there is still much exploration left to be done with transformers; these explorations may plant the seeds of or inspires what comes next. We hope that this survey provides an initial roadmap for these brave explorers. 123 That is, until his innovations get stolen. Steve Jobs is also reported to have said, "I'm going to destroy Android, because it's a stolen product. I'm willing to go thermonuclear war on this." 124 For this reason, we have taken care throughout this survey to not just cite the most recent (and conveniently locatable) reference for a particular idea, but to trace back its intellectual history. In some cases, this has involved quite extensive and interesting "side quests" involving consultations with senior researchers who have had firsthand knowledge of the work (e.g., worked in the same lab that the idea was developed)-in essence, oral histories. We are confident to differing degrees whether we have properly attributed various ideas, and welcome feedback by readers to the contrary. We believe it is important to "get this right".