ED 2 LM : Encoder-Decoder to Language Model for Faster Document Re-ranking Inference (2022)

Abstract: State-of-the-art neural models typically encode document-query pairs using cross-attention for re-ranking. To this end, models generally utilize an encoder-only (like BERT) paradigm or an encoder-decoder (like T5) approach. These paradigms, however, are not without flaws, i.e., running the model on all query-document pairs at inference-time incurs a significant computational cost. This paper proposes a new training and inference paradigm for re-ranking. We propose to finetune a pretrained encoder-decoder model using in the form of document to query generation. Subsequently, we show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference. This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference. Our experiments show that this new paradigm achieves results that are comparable to the more expensive cross-attention ranking approaches while being up to 6.8X faster. We believe this work paves the way for more efficient neural rankers that leverage large pretrained models.

## introduction leveraging transformer architecture to model the concatenation of a query-document pair is a well-established approach for document ranking @xcite . today, modern neural methods for re-ranking are based on the encoderonly (e.g., bert @xcite ) or encoderdecoder (e.g., t5 @xcite ) paradigm where query-document interactions are modeled by the encoder's attention mechanism. unfortunately, these paradigms are computationally prohibitive given that the model has to be run on all document-query pairs during inference. to this end, it is commonplace to use less powerful but computationally lightweight dual encoder models @xcite @xcite @xcite for first-pass retrieval and to only run the more expensive re-ranker on a small subset of retrieved candidates. even with this setup, cross-attention-based re-ranking can still be expensive, especially when larger pretrained transformer models are used. as such, this paper is primarily concerned with improving inference-time re-ranking efficiency while maintaining comparable effectiveness to existing cross-attention models. the novelty of this paper lies in a new paradigm for re-ranking that provides up to 6.8x speedup without any degradation in shallow-pool effectiveness. concretely, we propose a new method for inference-time decomposition of encoder-decoder architectures into decoder-only language models. given a pretrained sequence-to-sequence model, we finetune the encoder-decoder model using a document-to-query multi-task loss. at inference, we decompose the encoder-decoder architecture into a decoder-only language model (lm) that learns to interpret from a memory store of encoded document tokens representations using attention. the document-query pair score can be interpreted as the likelihood of generating the query given the encoded document term representations. there are multiple efficiency benefits to our proposed design. first, significant inference-time cost savings are unlocked since the document term memory store can be pre-computed in advance and acts as a read-only memory. second, our redesign also exploits the fact that queries are generally much shorter than documents. during inference time, only query tokens have to be passed through the decoder stack when attending to the pre-computed document representations which allows us to also obtain an additional speed advantage over encoder-only bert-like models. third, computing the query likelihood is computationally simple and does not require the typical costs associated with autoregressive generation models. the overall contributions of this work can be summarized as follows: a few studies that are most closely related to this work focus on leveraging the generative nature of pretrained encoder-decoder language models. a natural practice is to directly use the likelihood of generating the query given a document to rank the documents @xcite @xcite . however, these methods mostly perform substantially worse than cross-attention ranking models. another work @xcite transforms the likelihood of generating the query into a discriminative loss, where an "unlikelihood" loss is introduced for negative query-document pairs. despite relatively better performance than using vanilla maximum likelihood estimation (mle), we found that their method still underperforms cross-attention ranking models. our proposed method uses a combination of query generation loss and a cross-entropy loss on a specific token, which is capable of achieving comparable performance to cross-attention models. @xcite proposes query generation as an auxiliary task during training and shows improved performance. however, the proposed model still takes both a query and a document as input in the main ranking task and hence would be as costly as cross-attention ranking models during inference. finally, the recent differentiable search index @xcite proposes end-to-end ranking via text generation using an encoder-decoder t5 model. ## the proposed method this section describes the ed2lm model. see fig. 1 for an overview of the approach. ## experiment setup this section describes our experimental setup. ## results in this section, we examine the effectivenessefficiency trade-off of ed2lm on the passage reranking task. the results of t5, ed2lm, bert, and prettr have been displayed in ## ed2lm for question generation question generation has played an important role for different downstream tasks @xcite @xcite . we conjecture that the combination of generation and ranking loss used in ed2lm has the potential to improve question generation when compared to models trained with generation loss only. we evaluate this conjecture by comparing questions generated by vanilla generator trained with question likelihood only @xcite and ed2lm in different scenarios: manual inspection, assessment with automatic metrics and synthetic training data generation. for question generation task, an extra "eos" token (namely, the end of sequence token) is inserted between the question and the true/false token. our pilot experiments show that this change does not influence the ranking performance but boosts the generation quality of ed2lm. we adopt the top-k sampling decoding @xcite (set @xmath0. ## conclusion in this work, we propose a novel model named ed2lm. ed2lm encodes documents and decodes the query using a trailing binary class token appended to the query for ranking. by training on a dataset with more unique questions (namely, "rocketqa-merge" @xcite ) and optimizing both query likelihood and a discriminative loss over the true/false token, ed2lm achieves competitive results compared to corresponding t5 models. when used as a decoder-only language model during inference, ed2lm provides up to 6.8x speedup without sacrificing effectiveness. we further demonstrate that ed2lm could generate questions with less hallucination. for future works, we plan to investigate the uses of ed2lm for different (generation) tasks such as multi-sentence compression (mrc) @xcite , headline generation @xcite , and list question answering @xcite .