Modularized Transfomer-based Ranking Framework (2020)

Abstract: Recent innovations in Transformer-based ranking models have advanced the state-of-the-art in information retrieval. However, these Transformers are computationally expensive, and their opaque hidden states make it hard to understand the ranking process. In this work, we modularize the Transformer ranker into separate modules for text representation and interaction. We show how this design enables substantially faster ranking using offline pre-computed representations and light-weight online interactions. The modular design is also easier to interpret and sheds light on the ranking process in Transformer rankers.

## introduction neural rankers based on transformer architectures @xcite fine-tuned from bert @xcite achieve current state-of-theart (sota) ranking effectiveness @xcite . the power of the transformer comes from self-attention, the process by which all possible pairs of input tokens interact to understand their connections and contextualize their representations. self-attention provides detailed, token-level information for matching, which is critical to the effectiveness of transformer-based rankers @xcite . when used for ranking, a transformer ranker takes in the concatenation of a query and document, applies a series of self-attention operations, and outputs from its last layer a relevance prediction @xcite . the entire ranker runs like a black box and hidden states have no explicit meanings. this represents a clear distinction from earlier neural ranking models that keep separate text representation and distance (interaction) functions. transformer rankers are slow @xcite , and the black-box design makes it hard to interpret their behavior. we hypothesize that a transformerbased ranker simultaneously performs text representation and query-document interaction as it processes the concatenated pair. guided by this hypothesis, we decouple representation and interaction with a modualarized reranking system (mores). mores consists of three transformer modules: the document representation module, the query representation module, and the interaction module. the two representation modules run independently of each other. the document representation module uses self-attention to embed each document token conditioned on all document tokens. the query representation module embeds each query token conditioned on all query tokens. the interaction module performs attention from query representations to document representations to generate match signals and aggregates them through self-attention over query tokens to make a relevance prediction. by disentangling the transformer into modules for representation and interaction, mores can take advantage of the indexing process: while the interaction must be done online, document representations can be computed offline. we further propose two strategies to pre-compute document representations that can be used by the interaction module for ranking. our experiments on a large supervised ranking dataset demonstrate the effectiveness and efficiency of mores. it is as effective as a stateof-the-art bert ranker and can be up to 120× faster at ranking. a domain adaptation experiment shows that the modular design does not affect the model transfer capability, so mores can be used under low-resource settings with simple adaptation techniques. by adapting individual modules, we discovered differences between represen-tations and interaction in adaptation. the modular design also makes mores more interpretable, as shown by our attention analysis, providing new understanding of black-box transformer rankers. ## related work neural ranking models for ir proposed in previous studies can be generally classified into two groups @xcite : representation-based models, and interaction-based models. representation-based models learn latent vectors (embeddings) of queries and documents and use a simple scoring function (e.g., cosine) to measure the relevance between them. such methods date back to @xcite and classical siamese networks @xcite . more recent research considered using modern deep learning techniques to learn the representations. examples include dssm @xcite , c-dssm @xcite , etc. representations-based models are efficient during evaluation because the document representations are independent of the query, and therefore can be pre-computed. however, compressing a document into a single low-dimensional vector loses specific term matching signals @xcite . as a result, previous representation-based ranking models mostly fail to outperform interactionbased ones. interaction-based models, on the other hand, use a neural network to model the word-level interactions between the query and the document. examples include drmm @xcite and k-nrm @xcite . recently, transformers @xcite , especially bert @xcite based transformers, have been widely used in information retrieval ranking tasks @xcite @xcite . bert-based rankers concatenate query and document into a single string and apply self-attention that spans over the query and the document in every layer. rankers using pre-trained transformers such as bert has become the current state-ofthe-art @xcite . however, the performance gains come at the computational cost of inferring the many token-level interaction signals at the evaluation time, which scales quadratically to the input length. it is an open question whether we can combine the advantages of representation-based and interaction-based approaches. little research has studied this direction prior to this work. there are several research directions aiming to reduce the computational cost of transformer models. one line of research seeks to compress the big transformer into smaller ones using model pruning @xcite or knowledge distillation @xcite . another line of research aims to develop new transformer-like units that have lower complexity than the original transformer. for example, @xcite introduces sparse factorizations of the attention matrix which efficiently compute subsets of the attention matrix. the focus of this work is an efficient framework to combine transformers for ranking; all aforementioned techniques can be applied to individual transformers within our framework, and are therefore orthogonal to this paper. ## proposed method in this section, we introduce the modularized reranking system (mores), how mores can speed up retrieval, and how to effectively train and initialize mores. ## effectiveness and efficiency in supervised ranking the first experiment compares the effectiveness and efficiency of mores to a state-of-the-art bert ranker for supervised ranking. ## adaptation of mores and modules the second experiment uses a domain-adaptation setting to investigate whether the modular design of mores affects adaptation and generalization ability, and how the individual interaction and representation modules behave across domains. 7 this follows model dimension in bert ## analysis the modular design of mores allows representation and interaction to be inspected separately, providing better interpretability than a black-box transformer ranker. figure 2 examines the attention with mores for a hard-to-understand query "what is paranoid sc" where "sc" is ambiguous, along with a relevant document "paranoid schizophrenia is a psychotic disorder. in-depth information on symptoms...." 8 in the document representation module (figure 2a ), we can see that "disorder" uses "psychotic" and "schizophrenia" for contextualization, making itself more specific. in the query representation module (figure 2b ), because the query is short and lacks context, "sc" incurs a broad but less meaningful attention. the query token "sc" is further contextualized in the interaction module (figure 2c ) using information from the document side -"sc" broadly attends to the document token in the first ib to disambiguate itself. with the extra context, "sc" is able to correctly attend to "schizophrenia" in the second ib to produce relevance signals (figure 2d ). this example explains why mores 1× ib performs worse than mores with multiple ibsambiguous queries need to gather context from the document in the first ib before making relevance estimates in the second. more importantly, the example indicates that the query to document attention has two distinct contributions: understand query tokens with the extra context from the document, and match query tokens to document tokens, with the former less noticed in the past. we believe mores can be a useful tool for better interpreting and understanding sota black-box neural rankers. ## conclusion state-of-the-art neural rankers based on the transformer architecture consider all token pairs in a concatenated query and document sequence. though effective, they are slow and challenging to interpret. this paper proposes mores, a modular transformer ranking framework that decouples ranking into document representation, query representation, and interaction. mores is effective while being efficient and interpretable. experiments on a large supervised ranking task show that mores is as effective as a state-of-theart bert ranker. with our proposed document representation pre-compute and re-use methods, mores can achieve 120x speedup in online ranking while retaining accuracy. domain adaptation experiments show that mores' modular design does not hurt transfer ability, indicating that mores can be adapted to low-resource domains with simple techniques. decoupling representation and interaction provides new understanding of transformer rankers. complex full query-document attention in stateof-the-art transformer rankers can be factored into independent document and query representation, and shallow light-weight interaction. we further discovered two types of interaction: further query understanding based on the document, and the query to document tokens matching for relevance. moreover, we found that the interaction in ranking is less domain-specific, while the representations need more domain adaptation. these findings provide opportunities for future work towards more efficient and interpretable neural ir.