Document Ranking with a Pretrained Sequence-to-Sequence Model (2020)

Abstract: This work proposes the use of a pretrained sequence-to-sequence model for document ranking. Our approach is fundamentally different from a commonly adopted classification-based formulation based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance labels as “target tokens”, and how the underlying logits of these target tokens can be interpreted as relevance probabilities for ranking. Experimental results on the MS MARCO passage ranking task show that our ranking approach is superior to strong encoder-only models. On three other document retrieval test collections, we demonstrate a zero-shot transfer-based approach that outperforms previous state-of-the-art models requiring in-domain cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only architecture in a data-poor setting. We investigate this observation in more detail by varying target tokens to probe the model’s use of latent knowledge. Surprisingly, we find that the choice of target tokens impacts effectiveness, even for words that are closely related semantically. This finding sheds some light on why our sequence-to-sequence formulation for document ranking is effective. Code and models are available at pygaggle.ai.

## introduction a simple, straightforward formulation of ranking is to convert the task into a classification problem, and then sort the candidate items to be ranked based on the probability that each item belongs to the desired class. applied to the document ranking problem in information retrieval-where given a query, the system's task is to return a ranked list of documents from a large corpus that maximizes some ranking metric such as average precision or ndcg-the simplest formulation is to deploy a classifier that estimates the probability each document belongs to the "relevant" class, and then sort all the candidates by these estimates. deep transformer models pretrained with language modeling objectives, exemplified by bert @xcite , have proven highly effective in a variety of classification and sequence labeling tasks in nlp; @xcite are the first to demonstrate their effectiveness in ranking tasks. since it is impractical to apply inference to every document in a corpus with respect to a query, these techniques are typically applied to rerank a list of candidates. in a typical end-to-end system, these candidates are taken from the results of a keyword search based on a "classic" ir scoring function such as bm25 @xcite . this leads to the standard multi-stage pipeline architecture where first-stage retrieval is followed by reranking using one or more machine learning models @xcite . this architecture underlies nearly all transformerbased approaches to document retrieval today, for example, cedr @xcite , bert-maxp @xcite , birch @xcite , and parade @xcite . applying bert (and its variants) to document ranking can be characterized as a classificationbased encoder-only approach. in contrast, we explore the use of a sequence-to-sequence encoderdecoder architecture-specifically, t5 @xcite -to ranking, which requires a trick to coax relevance probabilities out of model-generated "target tokens". we show that in a data-rich setting, with sufficient training examples, our approach outperforms a classification-based encoder-only model. however, our sequence-to-sequence model appears to be far more data-efficient, significantly outperforming bert with few training examples in a data-poor setting. the main advantage of our approach is that by "connecting" fine-tuned latent representations of relevance to output target tokens, we can exploit the model's latent knowledge (e.g., of semantics, linguistic relations, etc.) that has been captured through pretraining. we describe probing experiments that attempt to verify our intuitions by deliberately altering the target tokens to capture different aspects of "semantic relatedness". the contribution of this work is to present a novel approach to document ranking using a pretrained sequence-to-sequence model. while ranking with classification-based encoder-only architectures (bert and variants) is commonplace today, we are the first to describe ranking with encoderdecoder architectures and articulate its advantages. additional ablation and contrastive experiments reveal new insights on fundamental differences between these two approaches, and our technique to probe model behavior by manipulating the output target tokens is also methodologically novel. ## seq2seq ranking the main idea behind the text-to-text transfer transformer (t5) by @xcite is to cast every natural language processing task-for example, machine translation, question answering, and classification-as feeding a sequence-to-sequence model some input text and training it to generate some output text. these include tasks that can be naturally viewed as "sequence in, sequence out" (e.g., machine translation) as well as tasks for which a sequence-to-sequence formulation seems unnatural (e.g., coreference resolution). the t5 architecture can be viewed as a natural progression of "vanilla transformers" by @xcite , but with pretraining inspired by bert's masked language model objective. like bert, a pretrained t5 model is then fine-tuned on various downstream tasks, where each task is associated with a specific "input template". for example, to translate text from english to german, the sentence to be translated is prefixed with the literal phrase "translate english to german:". we follow the same approach and formulate document ranking as a relevance prediction problem, i.e., the task is to estimate a relevance score that quantifies the extent to which a candidate document is relevant to a query. we devise the following input template to capture this task: query: [q] document: [d] relevant: after some amount of empirical exploration we arrived at an effective solution (see section 5.3 for more details). at inference time, to extract useful probabilities from the model, we apply a softmax only on the logits of the "true" and "false" tokens. in other words, we compute pr(@xmath0, d), as the probability assigned to the "true" token normalized in this manner. this estimate is interpreted as the relevance score for each query-document pair. each candidate document from first-stage retrieval is independently fed to the model, and the final document ranking is simply a permutation of the initial candidate documents based on these estimated probabilities in descending order. although this trick may seem obvious in retrospect, we are quite certain of its novelty-a lead author of the t5 paper @xcite , in personal communication, affirmed that the authors never tried anything along these lines before because there was no need for the tasks that they were tackling. note that t5 tokenizes sequences using the sen-tencepiece model @xcite , which might split a word into subwords. we choose target tokens ("true" and "false") that are represented as single words; thus, each class is represented by a single logit. in the case where target tokens are split in multiple subwords, we would need a method to aggregate their logits into a single score; we thought it best to avoid this complexity. our formulation naturally begs the question: why "true" and "false" as the target tokens? we discuss this question in section 5.4. however, as a preview, we find that the choice of target tokens has a large impact on effectiveness in some circumstances, and these experiments shed light on why t5 works well for document ranking. true to the original motivation of @xcite , we explore the transfer capabilities of t5 (recall, the model name stands for text-to-text transfer transformer) by experimenting with zeroshot document ranking on different datasets. to summarize, we fine-tune the model on the ms marco passage dataset and directly apply it on three other test collections commonly used by the information retrieval community. this requires a modification to rank long documents at inference time, which we describe below. finally, while our experiments only examine t5, we note that our method can be used with any other pretrained sequence-to-sequence model such as bart @xcite , mass @xcite , unilm @xcite , and pegasus @xcite . we leave explorations of these models for future work. ## results main results on the ms marco passage retrieval task are shown in results on robust04, core17, and core18 are shown in as we expect, effectiveness increases with larger models, but in all cases t5 improves over both a bag-of-words as well as a query expansion baseline. note that the latter is considered to be a strong baseline, even for pre-bert neural ranking models @xcite . in many cases, we notice that the effectiveness improvement of t5-large over t5-base is small; we investigate this curious finding further in section 5.2. ## related work as with natural language processing, the advent of deep learning has transformed the information retrieval community. prior to deep learning, researchers and practitioners mostly adopt the paradigm known as "learning to rank", which is heavily driven by manual feature engineering @xcite . for example, commercial web search engines are known to incorporate thousands of features (or more) in their models. the introduction of continuous vector space representations coupled with neural models was exciting as it provides a potential path away from the need for handcrafted features. well-known early neural ranking models include drmm @xcite , @xcite , knrm @xcite , and co-pacrr @xcite ; the literature is too vast for an exhaustive review here, and thus we refer readers to past overviews @xcite . interestingly, however, a meta-analysis by @xcite finds that without sufficient training data, these neural models still perform worse than well-tuned bag-of-words query expansion baselines. however, in the past year or so, we have witnessed a dramatic shift to ranking models based on bert, starting with @xcite . the current state of the art models @xcite @xcite serve as the points of comparisons in ## conclusion the main contribution of this paper is to introduce a novel generation-based approach to document ranking using pretrained sequence-to-sequence models. our models outperform a classification-based encoder-only approach, especially in the data-poor setting with limited training data. we attempt to explain these observations in terms of hypotheses about the knowledge that a model gains from pretraining vs. fine-tuning on task-specific data. these hypotheses are operationalized into target token probing experiments, where we demonstrate that the model appears to exploit knowledge from its ability to generate fluent natural language text. exactly how remains an open research question and the focus of ongoing work.