Improve Dense Passage Retrieval with Entailment Tuning (2024)

Abstract: Retrieval module can be plugged into many downstream NLP tasks to improve their performance, such as open-domain question answering and retrieval-augmented generation. The key to a retrieval system is to calculate relevance scores to query and passage pairs. However, the definition of relevance is often ambiguous. We observed that a major class of relevance aligns with the concept of entailment in NLI tasks. Based on this observation, we designed a method called entailment tuning to improve the embedding of dense retrievers. Specifically, we unify the form of retrieval data and NLI data using existence claim as a bridge. Then, we train retrievers to predict the claims entailed in a passage with a variant task of masked prediction. Our method can be efficiently plugged into current dense retrieval methods, and experiments show the effectiveness of our method.

## introduction information retrieval(ir) is the process of searching and matching relevant information for a given query. due to its effectiveness, ir has been integrated into a wide range of modern nlp solutions, especially knowledge-intensive tasks such as opendomain qa @xcite , fact verification @xcite and retrieval-augmented generation (rag) @xcite . with the development of pre-trained language models (plm), dense retrieval methods @xcite have demonstrated remarkable performance in ir tasks by matching queries and contexts using vector representations learned by plms. in such a way, texts can be retrieved based on semantic relevance, thus avoiding problems such as vocabulary mismatching and providing more useful information for downstream tasks @xcite . a key challenge to ir is the ambiguous definition of relevance @xcite . on the one question passages answer who first step on the moon? armstrong loves the moon since he learns to walk. armstrong became the first man to walk on the moon in 1969. armstrong is the first human step on the moon. ## preliminaries in this section, we introduce the preliminaries of current dense retrieval framework. task definition. given a query q and a collection of passages p, the goal of the dense retriever m is to retrieve k passages from p that are most relevant to q. passages in p are encoded using m and represented in the form of dense vectors. they are pre-calculated and stored in a vector database, organized using a index such as faiss @xcite . at inference time, query q is first encoded using m . then, the relevance score of query q and a passage p is calculated using a similarity function f , and k passages with highest scores are retrieved and returned: training of denser retriever m can be broadly classified into two stages, which are instruced as follows. pre-training. based on plms, dense retriever are further trained in an unsupervised manner on largescale corpus such as nq and msmarco. this retrieval-oriented pre-train aims to adapt plms to dense retrieval by encoding richer information into document embedding, using tasks such as inverse cloze prediction and maskedsalient spans. this stage is not necessary but can provide better initialzation for fine-tuning. fine-tuning. fine-tuning employs supervised training scheme using much smaller annotated retrieval data. the paired dataset d consists of triplets {(q i , p + i , p - i,1 , . . . , p - i,m )} n @xmath0, where q is the query, p + is a positive passage relevant to q and {p - 1 , . . . , p - m } are negative passages irrelevant to q. 11377 dense retriever is often trained with a constrastive loss funtion as follows: ## entailment tuning in this section, we introduce our method of tuning dense retrievers by enhancing the entailment relationship between query and retrieved passages. this method can be easily plugged into current dense retriever training pipeline before supervised contrastive finetuning. unified prompting. to use nli data to augment the entailment tuning process, we first unified the format of nli data and passage retrieval data. nli data consists of pairs of statements premise and hypothesis. we use the prompt "<@xmath1. passage retrieval data, on the other hand, consists of triples of (q, p + , p -), where p + stands for positive passages, and p - stands for negative passages. to fit passage retrieval data into the entailment prompt, the question q is first transformed into a narrative-form claim c in an information-variant manner. specifically, we use a set of rules to effectively convert q to c. we divide questions into six categories: when, why, who, where, does, how. for example, a question in the form of "when did ..." is then mapped into claim "there exists a known time when ...". then, p + is positioned @xmath2. in this way, the passage data (q, p + ) can be composited in the same format like other nli data and mixed together in training. ## conclusion in this work, we study the definition of relevance in retrieval, especially in the setting of dense retrieval for qa. we bring forward the connection between dense passage retrieval and nli through a information-invariant question-to-claim transformation trick. based on this perspecitve, we conduct logical-form analysis and find experimental evidences to validate its reasonability. we further design an effective and efficient method called entailment tuning which can be easily plugged in to current dense retriever training pipeline. empirical results on dense passage retrieval and downstream tasks including open-domain qa and rag proves the advantage of our methods.