Query Generation with External Knowledge for Dense Retrieval (2022)

Abstract: Dense retrieval aims at searching for the most relevant documents to the given query by encoding texts in the embedding space, requiring a large amount of query-document pairs to train. Since manually constructing such training data is challenging, recent work has proposed to generate synthetic queries from documents and use them to train a dense retriever. However, compared to the manually composed queries, synthetic queries do not generally ask for implicit information, therefore leading to a degraded retrieval performance. In this work, we propose Query Generation with External Knowledge (QGEK), a novel method for generating queries with external information related to the corresponding document. Specifically, we convert a query into a triplet-based template form to accommodate external information and transmit it to a pre-trained language model (PLM). We validate QGEK on both in-domain and out-domain dense retrieval settings. The dense retriever with the queries requiring implicit information is found to make good performance improvement. Also, such queries are similar to manually composed queries, confirmed by both human evaluation and unique & non-unique words distribution.

## introduction information retrieval (ir) is the task of collecting relevant documents from a large corpus when given a query. ir not only plays an important role in the search system by itself, but is also crucially applied to various nlp tasks such as open-domain qa @xcite and citation-prediction @xcite with its ability to find grounding documents. as the simplest retrieval method, traditional term-based sparse models such as tf-idf and bm25 @xcite are widely used. however, these sparse retrieval models are unable to capture the semantic similarities without explicit * corresponding author … that zebra mussels have also had an effect on fish populations … they were first detected in canada in the great lakes in 1988, … when did zebra mussels come to north america where are mussels located lexical overlaps between the query and its relevant documents. as a solution, dense retrieval models are recently proposed where query and document representations are embedded into the latent space @xcite , though they require a large amount of paired query-document training samples for notable performance, which is very challenging and expensive. in response, a zero-shot setting is often adopted, but dense retrievers are known to show poor performance on a new target domain @xcite @xcite . one possible solution is to generate synthetic queries by fine-tuning a pre-trained language model (plm) on a large ir benchmark dataset, and to use such queries for training dense retrievers @xcite @xcite . however, this method does not yet provide synthetic queries whose quality is comparable to that of human labeled ones, thus hindering retrieval performance. in particular, we argue that, for the effective training of dense retrievers, query samples should be allowed to contain external knowledge that is not explicitly shown in documents. as figure 1 [subject][name]call of duty: world at war[type]…[description]… [relation][name]…[type]… [object][name]playstation3[type]…[description]…game console… playstation3 … home video game console… what console is call of duty: world at war available on? triple-based template query template construction shows, the human labeled query contains the external knowledge that canada and north america are related, which is easily grasped by humans but not by the machine. also, unique words in the query, often considered as external knowledge, are more frequently included in the human labeled queries than in the synthetic queries. the dense retrievers would better capture semantic relations if they are trained with such queries that show more characteristics of human labeled ones. in this paper, we focus on generating queries with external knowledge by employing a simple method of explicitly transmitting document-related information to a plm. even though plms can handle hidden information to some extent by learning from a large amount of data, we argue that transmitting additional pieces of external knowledge to a plm contributes positively to generating queries requiring external knowledge. specifically, we first interpret the given query into a tripletbased template to consider the given document and related external knowledge together. a plm is then fine-tuned to generate queries from tripletbased templates, together with a processed kb-qa dataset. the dense retriever is trained with the synthetic queries from the template extracted from the given document and corresponding external knowledge. the proposed method, henceforth referred to as query generation with external knowledge (qgek), is schematically illustrated in figure 2 . we validate qgek in both in-domain and out-domain (zero-shot) dense retrieval settings with diverse evaluation metrics. the experimental results show that queries that require external knowledge to answer are helpful for improving retrieval performance. furthermore, we provide detailed qualitative analyses of synthetic queries and discuss which aspects of queries should be considered when training dense retrieval models. our contributions in this work are threefold: ## methods qgek is designed to generate a new synthetic query that requires an implicit inference process for the answer by exploiting both the given document and external knowledge hidden in the document. first, we interpret the query as the @xmath0,r,@xmath1, where the triplet is converted into a single-text template to simplify the transmission to a plm. then, we construct triplet-based template & query pairs as training datasets for fine-tuning a plm. for generating a query from target documents, the triplet-based template is extracted from a general document. ## experimental setups we evaluate the performances of the dense retriever when trained with the synthetic queries compared to the human labeled queries. the dense retriever used in our experiments is the dpr @xcite . the train dataset of the dense retriever is the pairs of the documents of natural question (nq) @xcite , also exploited as the source of the query generator, and the synthetic queries of the proposed method. ## conclusion we presented a novel query generation method, qgek, that generates synthetic queries in a form more similar to human labeled queries by using external knowledge. in order to use unprocessed external knowledge, we convert a query into a tripletbased template, which can include information of subjects and answers. remarkably, when dense retrieval models are trained with the queries generated from qgek, the performance has improved much compared to using the queries without external knowledge. also, we have shown that including external knowledge give rises to the distribution of the unique words similar to that of the human labeled queries. we believe that qgek can also be applied to the other generation methods by orthogonally adding some external knowledge processing modules. for future work, we plan to generate queries both close to human labeled ones and optimized for ir tasks and to allow the template to accept more general logical forms for diverse highquality queries. the code and data will be made available for public access.