M at R ank: Text Re-ranking by Latent Preference Matrix (2022)

Abstract: Text ranking plays a key role in providing content that best answers user queries. It is usually divided into two sub-tasks to perform efficient information retrieval given a query: text retrieval and text re-ranking. Recent research on pretrained language models (PLM) has demonstrated efficiency and gain on both sub-tasks. However, while existing methods have benefited from pre-trained language models and achieved high recall rates on passage retrieval, the ranking performance still demands further improvement. In this paper, we propose MatRank, which learns to re-rank the text retrieved for a given query by learning to predict the most relevant passage based on a latent preference matrix. Specifically, MatRank uses a PLM to generate an asymmetric latent matrix of relative preference scores between all pairs of retrieved passages. Then, the latent matrix is aggregated row-wise and column-wise to obtain global preferences and predictions of the most relevant passage in two of these directions, respectively. We conduct extensive experiments on MS MACRO, WikiAQ, and SemEval datasets. Experimental results show that MatRank has achieved new state-of-the-art results on these datasets, outperforming all prior methods on ranking performance metrics.

## introduction the enormous growth of online content has posed significant challenges for information retrieval (ir) systems, e.g., google, baidu, bing, and quora. ranking methods are crucial in ir systems in identifying the most relevant passage and document to answer a user query. generally, text ranking is divided into two steps, i.e., text retrieval and text re-ranking. in the text retrieval step, a number of passages or documents of high relevance to a user query are retrieved to ensure the efficiency of the text ranking. in the re-ranking step, all the retrieved paragraphs are ranked in decreasing order of relevance to the given query. recent studies adopt pre-trained language models (plms) to boost the ranking performance of ir systems. current state-of-the-art text retrieval techniques can provide decent performance in recalling the most relevant content of a query @xcite @xcite @xcite . however, how to better present those retrieved texts in order of their relevance to the query is still challenging. previous learning to rank methods can be roughly divided into three categories: 1) the point-wise methods (e.g., mcrank @xcite ), which score each candidate independently, 2) the pair-wise methods (e.g., ranknet @xcite , lambdarank @xcite ), which employ a pair-wise loss between every pair of candidate paragraphs, and 3) list-wise methods (e.g., lambdamart @xcite , listrank-mf @xcite ), most of which try to rank all candidates at once. however, the sizes of the benchmarks are increasing due to the enormous growth of online texts. such large sizes have limited the number of annotations per query @xcite . that is, only a few truth answers are available for each query, rather than a ground-truth rank list. therefore, directly implementing the classical pair-wise or list-wise ranking methods on these benchmarks is infeasible due to the lack of groundtruth labels. moreover, the increasing online text brings more homogeneous passages, making the re-ranking even more challenging. to tackle the above challenges, recent research on text re-ranking @xcite @xcite @xcite @xcite utilizes plms to better understand the content of text before evaluating its relevance to a query. in case of limited annotations, they try to convert the text re-ranking problem into a classification task, which learns to predict the best relevant candidate directly. the outputs for each candidate are their probability of becoming the best choice, which is used to represent their relevance score to the query. however, such classification tasks do not utilize preference comparisons between candidates @xcite . furthermore, the limitation on the sequence length of the plms makes the re-ranking of long documents even harder. in this paper, we introduce the matrank (text re-ranking by latent preference matrix), an effective method for re-ranking the retrieved text by learning a latent preference matrix, which enables the preference assessment during the re-ranking process and makes full use of the annotation. given a query, matrank employs a plm to encode every two candidates into latent vectors and an mlp selector to generate a latent preference matrix, where each entry represents a preference score between two candidates. the matrix entries are then aggregated to generate the row-wise preference and the column-wise preference, which are used to predict the rank result. the entire model is trained endto-end by simultaneously minimizing the list-wise classification losses on both the row-wise preference and the column-wise preference to identify their winners, which are used to match the ground truth. specifically, we make the following contributions: we proposed the matrank method, which can implicitly assess the preference between every two candidates by generating a latent preference matrix even when the preference labels are unavailable. through the aggregation of the estimated pair-wise preference scores in the matrix, the matrank can also make use of the list-wise information to generate a better ranking result. to handle the re-ranking for long documents, we design matrank-split, which splits the long documents into pieces and uses matrank to generate a latent preference matrix of these text pieces. each document will be represented by its text piece with the highest global preference. with such a design, our matrank-split can aggregate much more information from a document to assess the preference between every two documents more accurately. to demonstrate the performance of the proposed method, we have tested matrank on the ms macro @xcite passage, wikiqa @xcite , and three semeval datasets @xcite @xcite . the results indi-cate that the proposed method significantly outperforms a wide range of existing passage re-ranking methods. furthermore, experimental results on the ms macro document dataset demonstrate the superiority of the proposed matrank-split model over state-of-the-art non-ensemble methods on longdocument re-ranking tasks. we have also performed extensive ablation studies to demonstrate the effectiveness of the proposed model design of both the matrank and the matrank-split models by exploring a variety of potential variants. ## related work in this section, we introduce the related works in the following aspects. ## methodology in this section, we introduce the proposed matrank, which learns to rank based on the content comparison between different candidates, in detail. we will mainly discuss the design parts of matrank based on the ms macro datasets. given a query q and the corresponding retrieved candidate list @xmath0, the goal of text reranking is to generate a rank list @xmath1, where m t is the number of candidates and m r (≤ m t ) represents the length of the generated rank list. here, we assume m @xmath2. ## experiments we develop and test our method on the ms macro passage and the ms macro document datasets, which are widely used benchmarks in many recent studies on passage ranking @xcite @xcite . to further justify the robustness of the matrank, we also conduct experiments on other public datasets such as the wikiqa @xcite and the semeval datasets @xcite @xcite . detailed descriptions of those datasets are given in appendix a. we adopt the same settings and retrievers as the current state-of-the-art methods on each dataset for fair comparisons. we report the mrr (mean reciprocal rank) and, if applicable, the map (mean average precision) as the evaluation metrics. we fine-tune our model on 8 nvidia tesla a100 gpus (with 40g ram) under fp16. we employ the adafactor @xcite optimizer with a learning rate of 1e-5 and a 10% warmup ratio to train our models on all the datasets. the epochs are set to be 3 on the ms marco dataset and 5 on the wikiqa and the semeval datasets. the positive to the hard negative ratio is set to be 1:96 on the ms marco passage dataset, 1:16 on the ms marco document dataset, and 1:4 on the wikiqa and the semeval datasets. in addition, we set the max text piece of document re-ranking to 5 and the max length of each piece to 512. our model contains 120 million parameters. the implementation details of the compared baselines are given in appendix b. ## conclusion in this paper, we introduce matrank for passage re-ranking. matrank adopts a pre-trained language model to generate an asymmetric latent matrix of relative preference scores, which is aggregated rowwise and column-wise to predict the most relevant passage in both directions, respectively. the model is trained in an end-to-end manner with classifi-cation loss. by minimizing the list-wise classification loss generated from the pair-wise comparison scores, matrank makes full use of all the retrieved passages, even with only a few annotations. furthermore, we propose matrank-split for long document re-ranking. in matrank-split, the documents are split into text pieces to fit the comparison method designed in matrank. experimental results on several benchmark datasets demonstrate the effectiveness of both the matrank and matrank-split models.