BERT-QE : C ontextualized Q uery E xpansion for D ocument R e-ranking (2020)

Abstract: Query expansion aims to mitigate the mismatch between the language used in a query and in a document. However, query expansion methods can suffer from introducing non-relevant information when expanding the query. To bridge this gap, inspired by recent advances in applying contextualized models like BERT to the document retrieval task, this paper proposes a novel query expansion model that leverages the strength of the BERT model to select relevant document chunks for expansion. In evaluation on the standard TREC Robust04 and GOV2 test collections, the proposed BERT-QE model significantly outperforms BERT-Large models.

## introduction in information retrieval, the language used in a query and in a document differs in terms of verbosity, formality, and even the format (e.g., the use of keywords in a query versus the use of natural language in an article from wikipedia). in order to reduce this gap, different query expansion methods have been proposed and have enjoyed success in improving document rankings. such methods commonly take a pseudo relevance feedback (prf) approach in which the query is expanded using topranked documents and then the expanded query is used to rank the search results @xcite @xcite . due to their reliance on pseudo relevance information, such expansion methods suffer from any non-relevant information in the feedback documents, which could pollute the query after expansion. thus, selecting and re-weighting the information pieces from prf according to their relevance before re-ranking are crucial for the effectiveness of * this work has been done before joining amazon. the query expansions. existing works identify expansion tokens according to the language model on top of feedback documents, as in rm3 @xcite , extract the topical terms from feedback documents that diverge most from the corpus language model @xcite , or extract concepts for expansion @xcite . in the context of neural approaches, the recent neural prf architecture @xcite uses feedback documents directly for expansion. all these methods, however, are under-equipped to accurately evaluate the relevance of information pieces used for expansion. this can be caused by the mixing of relevant and non-relevant information in the expansion, like the tokens in rm3 @xcite and the documents in nprf @xcite ; or by the facts that the models used for selecting and re-weighting the expansion information are not powerful enough, as they are essentially scalars based on counting. inspired by the recent advances of pre-trained contextualized models like bert on the ranking task @xcite , this work attempts to develop query expansion models based on bert with the goal of more effectively using the relevant information from prf. in addition, as indicated in previous studies @xcite , the (pre-)trained bert-based ranking models have a strong ability to identify highly relevant chunks within documents. this actually provides advantages in choosing text chunks for expansion by providing more flexibility in terms of the granularity for expansions, as compared with using tokens @xcite , concepts with one or two words @xcite , or documents @xcite . given a query and a list of feedback documents from an initial ranking (e.g., from bm25), we propose to re-rank the documents in three sequential phases. in phase one, the documents are re-ranked with a fine-tuned bert model and the top-ranked documents are used as prf documents; in phase two, these prf documents are decomposed into text chunks with fixed length (e.g., 10), and the relevance of individual chunks are evaluated; finally, to assess the relevance of a given document, the selected chunks and original query are used to score the document together. to this end, a novel query expansion model, coined as bert-qe, based on the contextualized model is developed. contributions of this work are threefold. 1) a novel query expansion model is proposed to exploit the strength of contextualized model bert in identifying relevant information from feedback documents; 2) evaluation on two standard trec test collections, namely, robust04 and gov2, demonstrates that the proposed bert-qe-lll could advance the performance of bert-large significantly on both shallow and deep pool, when using bert-large in all three phases; 3) we further trade-off the efficiency and effectiveness, by replacing bert-large with smaller bert architectures and demonstrate that, with a smaller variant of bert-qe, e.g., bert-qe-lmt, one could outperform bert-large significantly on shallow pool with as least as an extra 3% computational cost; meanwhile, a larger variant, e.g., bert-qe-lls, could significantly outperform bert-large on both shallow and deep pool with 30% more computations. ## method in this section we describe bert-qe, which takes a ranked list of documents as input (e.g., from an unsupervised ranking model) and outputs a re-ranked list based on the expanded query. ## experimental setup in this section, we describe our experiment configurations. source code, data partitions for crossvalidation, result files of initial rankings, and the trained models are available online foot_0 . ## results in this section, we report results for the proposed bert-qe model and compare them to the baseline models. first, in section 4.1, we use bert-large models for all three phases of bert-qe. in section 4.2, we evaluate the impact of using smaller bert models ( ## conclusion this work proposes a novel expansion model, coined as bert-qe, to better select relevant information for query expansion. evaluation on the robust04 and gov2 test collections confirms that bert-qe significantly outperforms bert-large with relatively small extra computational cost (up to 30%). in future work, we plan to further im-prove the efficiency of bert-qe, by combining the proposed bert-qe with the pre-computation techniques proposed recently @xcite , wherein most of the computations could be performed offline.