Virtual Knowledge Graph Construction for Zero-Shot Domain-Specific Document Retrieval (2022)

Abstract: Domain-specific documents cover terminologies and specialized knowledge. This has been the main challenge of domain-specific document retrieval systems. Previous approaches propose domain-adaptation and transfer learning methods to alleviate this problem. However, these approaches still follow the same document representation method in previous approaches; a document is embedded into a single vector. In this study, we propose VKGDR. VKGDR represents a given corpus into a graph of entities and their relations (known as a virtual knowledge graph) and computes the relevance between queries and documents based on the graph representation. We conduct three experiments 1) domain-specific document retrieval, 2) comparison of our virtual knowledge graph construction method with previous approaches, and 3) ablation study on each component of our virtual knowledge graph. From the results, we see that unsupervised VKGDR outperforms baselines in a zero-shot setting and even outperforms fully-supervised bi-encoder. We also verify that our virtual knowledge graph construction method results in better retrieval performance than previous approaches.

## introduction in domain-specific qa, building retrievers is challenging since queries and documents in a specific domain cover terminologies and specialized knowledge, which are not well covered in general documents. @xcite @xcite . another problem is the difficulty in building datasets for training retrievers. this problem comes from 1) the complexity of knowledge treated in the documents, and 2) costly dataset maintenance; recall that domain-specific documents are frequently updated (e.g., software manuals are updated whenever there is a version update) @xcite @xcite . recent domain-specific document retrieval studies propose domain-adaptation and transfer learning methods @xcite @xcite @xcite . however, these methods still use the conventional document representation method, embedding a document into a single vector. this is problematic because a single vector is insufficient to cover complex knowledge in a domain-specific document. semi-structured knowledge representation methods effectively address this problem, but they have only been applied to open-domain documents. @xcite @xcite @xcite . in this paper, we propose an automatic virtual knowledge graph construction method for zeroshot domain-specific document retrieval. a virtual knowledge graph (vkg) is a graph representation of a corpus that consists of entities and their relations. in vkg, the relations are represented by relation vectors @xcite . this semi-structured representation enables explicit reasoning over the corpus. we apply this framework to domain-specific document retrieval. one of the key components of the vkg construction method is a relation encoder, which computes relation vectors of two entities. this study shows that previous supervision methods for relation encoders are insufficient for domain-specific documents, and we propose a novel distant-supervision method. we validate vkgdr in three types of experiments. first, we conduct zero-shot domain-specific document retrieval on two domain-specific qa datasets: techqa @xcite and photo-shopquia @xcite . the results show that vkgdr outperforms domain-adaptation and transfer learning methods. from this experiment, we also verify that unsupervised vkgdr outperforms a fully-supervised dense retriever. second, we show that our distant-supervision method for training the relation encoder outperforms previous approaches. in this experiment, we construct vkgs with our relation encoders and baselines' encoders. then, we measure the retrieval performance of each vkg. third, we conduct an ablation study on two main components of a vkg, graph representation of a corpus and relation vectors. the results show that each component increases the retrieval performance of vkgdr by a large margin. ## related work a virtual knowledge graph is a graph representation of a corpus that consists of entities and their relations. the relations of entities are represented by relation vectors. @xcite propose a differentiable vkg for multi-hop qa. their vkg is trained by the end-to-end supervision method on question-answer pairs. @xcite use vkg for knowledge graph qa. they apply relation encoders used in relation extraction studies to vkg construction and follow distant-supervision proposed by @xcite . our work provides a novel distant-supervision method for building a virtual knowledge graph for domain-specific documents. in section 5.1, we compare our methods with @xcite to validate the efficacy of our method. domain-specific documents cover complex knowledge and require advanced representation methods. previous approaches in domain-specific document retrieval focus on a document encoder training method and data scarcity problem but still follow conventional document representation methods. @xcite ; @xcite augment domain-specific question-answer pairs from an external corpus and train their encoders on the dataset. ## method we propose a novel domain-specific document retrieval method, vkgdr, based on a virtual knowledge graph (vkg). vkgdr consists of two modules: a vkg construction module and a document retrieval module. a vkg is a graph representation of a given corpus that connects mentions with directed edges, and each directed edge has a relation vector. @xcite . the document retrieval module computes the similarity between queries and documents with the mention pairs and their relation vectors. we describe notations and details of each module in the following sections. ## experimental setup we validate domain-specific document retrieval performance of vkgdr in three experimental settings. in the first experiment, we evaluate vkgdr and baselines in a zero-shot setting. the zeroshot setting emulates the real-world problem of domain-specific document retrieval; training data is insufficient or absent. additionally, we conduct the same experiment in a fully-supervised setting and show the efficacy of vkgdr. ## results in this section, we verify the efficacy of vkgdr with the experiments described in the previous section. the experimental results demonstrate three findings. first, vkgdr outperforms baselines in a zero-shot setting and a fully-supervised setting. furthermore, vkgdr without fine-tuning outperforms a fully-supervised bi-encoder. second, our distant-supervision method for the relation encoder outperforms the previous method. third, the two main components of vkgdr, graph representation of a corpus and relation vectors, are essential to achieve the zero-shot performance of vkgdr. we describe details of the experimental results in the following sections. ## conclusion the main challenge in domain-specific document retrieval is the difficulty of specialized knowledge and terminologies appearing in the documents. in this study, we propose vkgdr to resolve this problem. vkgdr consists of two modules: 1) the model that represents a given corpus into a graph of mentions and their relations and 2) a document retriever based on the vkg. we showed that vkgdr outperforms previous retrievers in zeroshot domain-specific document retrieval. when insufficient training data is provided, unsupervised vkgdr shows even better results than a fullysupervised dense retriever. also, we compared our vkg construction method with a previous method and showed that our method performs better on domain-specific documents.