GNN -encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval (2022)

Abstract: Recently, retrieval models based on dense representations are dominant in passage retrieval tasks, due to their outstanding ability in terms of capturing semantics of input text compared to the traditional sparse vector space models. A common practice of dense retrieval models is to exploit a dual-encoder architecture to represent a query and a passage independently. Though efficient, such a structure loses interaction between the query-passage pair, resulting in inferior accuracy. To enhance the performance of dense retrieval models without loss of efficiency, we propose a GNN-encoder model in which query (passage) information is fused into passage (query) representations via graph neural networks that are constructed by queries and their top retrieved passages. By this means, we maintain a dual-encoder structure, and retain some interaction information between query-passage pairs in their representations, which enables us to achieve both efficiency and efficacy in passage retrieval. Evaluation results indicate that our method significantly outperforms the existing models on MSMARCO, Natural Questions and TriviaQA datasets, and achieves the new state-of-the-art on these datasets.

## introduction large-scale query-passage retrieval is a core task in search systems, which aims to rank a collection of passages based on their relevance with regard to a query. to balance efficiency and effectiveness, existing work typically adopts a two-stage retrieval pipeline @xcite . the first-stage aims to retrieve a subset of candidate passages by a recall model from the entire corpus and the second stage aims to rerank the retrieved passages. in the first-stage retrieval, traditional approaches @xcite implemented term-based retriever (e.g. tf-idf and bm25) by weighting terms based on their frequency, which have limitations on representing semantics of text. recently, dense passage retrieval is drawing more and more attention in the task of passage retrieval @xcite . the underlying idea is to represent both queries and passages as embeddings, so that the semantic relevance can be measured via embeddings similarity. with the great success of pre-trained language models (plms) such as bert/roberta @xcite in natural language processing tasks, dense retrieval models parameterized by plms is emerging as the new state-of-the-art in a variety of passage retrieval tasks @xcite . two paradigms based on fine-tuned language models are typically built for retrieval: crossencoders and dual-encoders. typical crossencoders need to recompute the representation of each passage in the corpus once a new query comes, which is difficult to deploy in real-world search systems. in contrast, dual-encoders remove querypassage interaction by representing a query and a passage independently through two separate encoders (siamese encoders). hence, passage embeddings can be pre-computed offline, and online latency can be greatly reduced. thanks to this advantage, dual-encoders are more widely adopted in real-world applications. on the other hand, independent encoding without any interaction causes severe retrieval performance drop due to information loss. to improve the performance of dual-encoders, some efforts have been made to incorporate more complicated structures (i.e., late interaction) such as attention layers @xcite , the sum of maximum similarity computations @xcite , and the transformer layers @xcite into encoding. these late interaction strategies bring considerable improvements on retrieval performance but also increase computational overhead. moreover, interaction information is still neglected in earlier encoding of query and passage. in this work, we aim to achieve both efficiency and effectiveness in passage retrieval. the key idea is to maintain two independent encoders, and keep as much interaction information as possible in the meanwhile. to this end, we propose a novel approach that explicitly fuses query (passage) information into passage (query) embeddings through a graph neural network (gnn), and name the model gnn-encoder. our model is built upon the dualencoder, and learns query-interactive passage representations and passage-interactive query representations through a graph neural network. specifically, given a query set, we retrieve top passages for each query, and form a graph whose nodes are the queries and the passages, and edges reflect correspondence between query-passage pairs (i.e., if a passage is retrieved by the query). then, we initialize the gnn model with the representations of the pre-trained dual-encoder and cross-encoder, and then perform information propagation on the graph. to avoid information leakage, we further design a new training algorithm and name it masked graph training (mgt), in which the query set used for training gnn is no longer used to construct the query-passage graph in each training epoch. finally, the passage embeddings could be pre-computed offline corresponding to the gnn. thus our model holds the efficiency advantage inherited from the dual structure, and at the same time takes query-passage interaction into account. our contributions can be summarized as follows: our work touches on two strands of research within dense passage retrieval and graph neural network. ## conclusion in this paper, we introduced gnn-encoder for passage retrieval tasks and demonstrated its effectiveness on msmarco, natural questions and triv-iaqa datasets. the existing dual-encoder architecture, although very efficient, ignores interaction during passage (query) encoding due to its independent architecture. therefore we attempted to fuse query (passage) information into passage (query) representations via graph neural networks and maintain online efficiency of the dual-encoder. however, we may retrieve irrelevant passages for queries by dual-encoder when constructing the query-passage graph, which introduces noise in information propagating. hence we utilize gat layers and filter gate to reduce the noise, which are proved necessary by our various experiments. in the future, we will explore how to fuse more interaction information into gnn structure.