FAA : Fine-grained Attention Alignment for Cascade Document Ranking (2023)

Abstract: Document ranking aims at sorting a collection of documents with their relevance to a query. Contemporary methods explore more efficient transformers or divide long documents into passages to handle the long input. However, intensive query-irrelevant content may lead to harmful distraction and high query latency. Some recent works further propose cascade document ranking models that extract relevant passages with an efficient selector before ranking, however, their selection and ranking modules are almost independently optimized and deployed, leading to selecting error reinforcement and sub-optimal performance. In fact, the document ranker can provide fine-grained supervision to make the selector more generalizable and compatible, and the selector built upon a different structure can offer a distinct perspective to assist in document ranking. Inspired by this, we propose a fine-grained attention alignment approach to jointly optimize a cascade document ranking model. Specifically, we utilize the attention activations over the passages from the ranker as fine-grained attention feedback to optimize the selector. Meanwhile, we fuse the relevance scores from the passage selector into the ranker to assist in calculating the cooperative matching representation. Experiments on MS MARCO and TREC DL demonstrate the effectiveness of our method.

## introduction document ranking aims at ranking the candidate documents according to their relevance to an input query, and it has been widely applied in many natural language processing (nlp) and information retrieval tasks, such as search engines @xcite and question answering @xcite . due to the powerful representation ability of large-scale pre-trained language models (plms) (e.g., bert @xcite and figure 1 : the case of scope hypothesis. in this example, p 2 is strongly relevant to the query, and p 3 is weakly relevant where other passages focus on other topics different from query. roberta @xcite ) that have achieved impressive performance in a large number of nlp tasks, several researchers have considered making use of pre-trained models for document ranking @xcite @xcite . one major challenge in applying plms for neural document ranking is their difficulty in handling long texts due to high computational complexity and memory requirements, such as the 512 token limit for bert. in fact, documents typically contain long text, for example, the mean length of documents in 2019 trec deep learning track document collection is 1600 @xcite . to address this issue, various studies have been conducted to develop more efficient attention mechanisms in transformers @xcite , by simply truncating the document to meet the requirement for the relevance model @xcite , or by breaking down the long document into smaller segments or passages that can be processed individually by the pre-trained models @xcite @xcite . actually, long documents often contain a vari-ety of subjects, as evidenced by the scope hypothesis @xcite from traditional information retrieval. an illustration from the ms marco dataset @xcite is presented in figure 1 , and it is noted that only a small part of the document (e.g., p 2 and p 3 ) is relevant to the given query and different parts may be unequally informative to the query. thus even though existing techniques for modeling long documents have been demonstrated to be effective and efficient, utilizing the entire document can result in high query latency and intensive query-irrelevant content can be a distraction and negatively impact performance. consequently, some recent studies propose cascade document ranking models @xcite @xcite that extract relevant passages with an efficient selector before performing the ranking. however, their selection and ranking modules are almost independently optimized and deployed, leading to selecting error reinforcement and sub-optimal performance. moreover, these models do not differentiate between the passages or segments taken from a document while matching with the query. in fact, the document ranker can provide finegrained supervision to enhance the generalizability and compatibility of the selector. conversely, the selector, built upon a heterogeneous structure, can offer a distinct perspective to assist in document ranking. taking inspiration from this, we propose a fine-grained attention alignment approach (faa) to jointly optimize a cascade document ranking model. specifically, we initialize the passage selector as an efficient dual encoder and the document ranker with an effective crossencoder. to better optimize and make use of both worlds, we leverage the attention activations over the passages from the ranker as fine-grained attention feedback to optimize the selector. simultaneously, we incorporate the relevance scores from the passage selector into the ranker to assist in calculating the final cooperative matching representation. we conduct experiments on three public benchmarks: ms marco @xcite , trec-dl 2019 @xcite , and trec-dl 2020. the evaluation results show that our proposed model is better than several competitive baselines and our faa can bring significant improvement to the cascade model. to sum up, our contribution is three-fold: ## related works neural models for document ranking in the early stages, traditional algorithms like bm25 @xcite and tf-idf were commonly employed for ranking documents in information retrieval. with the development of neural network technology @xcite , some neural-based ranking models have been proposed @xcite @xcite @xcite . xiong et al. (2017) proposed a kernelbased neural ranking model (k-nrm) which used a kernel-pooling layer to combine word pair similarities with distributed representations. dai et al. (2018) extended k-nrm to conv-knrm which used convolutional neural networks to model ngram embedding. hofstätter et al. (2020b) proposed a transformer-kernel model which used a small number of transformer layers to contextualize query and document sequences independently and distilled the interactions between terms. compared to traditional methods, neural ranking models produce a dense representation of the queries and documents which improves the ranking performance. pre-trained models for document ranking recently a large number of transformer-based pretrained models have been proposed @xcite @xcite and shown their effectiveness in natural language processing tasks. therefore many works have utilized pre-trained models in document ranking tasks. @xcite used a sequence-to-sequence transformer model with document terms as input and produced the possible questions that the document might answer to expand document for document retrieval. finally this work used bert to re-rank these retrieved documents. @xcite used a pre-trained bert model to classify sentences into three categories and then fine-tuned the model using a point-wise ranking method for ranking documents. passage-level document ranking since the high demand for memory space and computing resources, pre-trained models usually have a limit on the input length, and the length of actual long documents is always beyond this limitation. to this end, some works proposed to split the long documents into multiple passages which satisfy the limitation of the input length of the pre-trained models @xcite @xcite . the studies applied pre-trained models to each passage individually and then combined the relevance scores at the passage level to generate the relevance scores for the entire document. for example, @xcite determined the relevance score of the document by utilizing the score of the first passage, the top-performing passage, and the summation of all passages, respectively. fu et al. (2022) proposed a multi-view interpassage interaction based ranking model (mir) with intra-passage attention and inter-passage attention, and used a multi-view aggregation layer to produce the document-level representation across multiple granularities. these works took all passages into document ranking which may introduce noise from the query-irrelevant passages and increase the query latency. to address this problem, some works proposed to pre-select query-relevant passages from all passages before aggregating. in this work, we propose a cooperative distillation and representation cascade ranking model which uses an efficient model as a passage selector to calculate passage-level relevance scores and select top-k passages, while uses an effective model as the document ranker to calculate the document-level relevance scores with the selected passages. ## methodology in this section we first formalize the document ranking task, then we introduce the model architecture and the proposed fine-grained attention alignment (faa) approach for model optimization. task formalization given a query q and a set of candidate documents @xmath0, d 2 , ..., d m } including both the ground-truth document and negative documents, where m is the number of the candidate documents, the task is to train a document ranking model r(q, d) with the training data d. when provided with a new query and its corre-sponding candidate documents, the ranking model assesses the relevance between the query and each candidate document by computing relevance scores. subsequently, it can arrange the documents in order based on these scores. model overview inspired by previous work on passage-level evidence for document ranking @xcite , in this paper we adopt the efficient and effective cascade document ranking paradigm which first extracts relevant passages with an efficient selector and then performs the ranking with a smart document ranker based on the pruned content. to better optimize and make use of both worlds, we propose a finegrained attention alignment approach to jointly optimize a cascade document ranking model. specifically, we utilize the attention activations over the passages from the ranker as fine-grained attention feedback to optimize the selector. additionally, in the process of document ranking, the passage-level relevance scores in the selector are fused in the document ranker to produce the cooperative matching representation for calculating the final matching score. by this means, the document ranker can provide fine-grained supervision to make the selector more generalizable and compatible, and the selector built upon a heterogeneous structure can conversely offer a distinct view to help the ranker. figure 2 presents the high-level architecture of the proposed method. ## experiments in this section, we first introduce the datasets we use, the evaluation metrics, the baselines, and the implementation details of our experiment. then we introduce the evaluation results and further analysis of our method. ## conclusion in this work, we propose faa, a cascade ranking model with a fine-grained attention alignment and cooperative matching representation. our model utilizes the fine-grained attention alignment approach to train the passage selector and fuses the passage-level relevance scores into document ranking to produce cooperative matching representation. the evaluation results on ms marco and trec dl demonstrate the effectiveness of faa.