Dense Passage Retrieval: Is it Retrieving? (2024)

Abstract: Large Language Models (LLMs) internally store repositories of knowledge. However, their access to this repository is imprecise and they frequently hallucinate information that is not true or does not exist. A paradigm called Retrieval Augmented Generation (RAG) promises to fix these issues. Dense passage retrieval (DPR) is the first step in this paradigm. In this paper, we analyze the role of DPR fine-tuning and how it affects the model being trained. DPR fine-tunes pre-trained networks to enhance the alignment of the embeddings between queries and relevant textual data. We explore DPR-trained models mechanistically by using a combination of probing, layer activation analysis, and model editing. Our experiments show that DPR training decentralizes how knowledge is stored in the network, creating multiple access pathways to the same information. We also uncover a limitation in this training style: the internal knowledge of the pre-trained model bounds what the retrieval model can retrieve. These findings suggest a few possible directions for dense retrieval: (1) expose the DPR training process to more knowledge so more can be decentralized, (2) inject facts as decentralized representations, (3) model and incorporate knowledge uncertainty in the retrieval process, and (4) directly map internal model knowledge to a knowledge base.

## introduction in just a few years, large language models (llms) have emerged from research labs to become a tool utilized daily by hundreds of millions of people and integrated into a wide variety of businesses. despite their popularity, these models have been critiqued for frequently hallucinating, confidently outputting incorrect information @xcite . such inaccuracies not only mislead people but also erode trust in llms. trust in these systems is crucial to their success and rate of adoption. the retrieval augmented generation @xcite paradigm is an approach to address hallucinations @xcite . unlike traditional llm interactions where a query directly prompts an output from the model, an intermediary step is introduced. initially, a "retrieval" model processes the query to gather additional information from a knowledge base, such as wikipedia or the broader internet. this additional information alongside the original query is fed to the llm, increasing the accuracy of the answers that the llm generates. for this paradigm to be effective, the underlying retrieval model has to excel at finding accurate and relevant information. typically, model performance is evaluated based on metrics that consider the top-5, top-20, top-50, and top-100 retrieved passages. however, recent studies indicate that llms predominantly use information from the top-1 to top-5 passages, underscoring the importance in rag of not only high recall in retrieval but also precision in ranking @xcite . one approach to achieve both high recall and precision involves integrating a "reranking" model, which adjusts the order of retrieved passages to improve the relevance of the top-ranked passages @xcite . however, this approach adds the computational and maintenance cost of an additional model to the pipeline and can also introduce errors. the alternative option is to improve retrieval models to directly retrieve and rank passages well. retrieval methods can be broadly categorized into two types: sparse and dense @xcite . sparse methods encode queries and passages into sparse vectors, usually based on terms that appear in the queries and passages @xcite . dense methods employ language models to encode the semantic information in queries and passages into dense vectors @xcite . dense methods often share two common properties: (a) the joint training of two or more encoding models -one for embedding a query and the other for embedding a knowledge base, and (b) contrastive training. these commonalities were introduced in the dpr method, inspiring many subsequent methods in the literature. in this paper, we analyze the original dpr method using the bert-base backbone. through the experiments in each section, we analyze dpr from multiple perspectives to understand what is changing in the backbone model during the training process. we find that: ## knowledge consistency between untrained and trained model language models are known to store a vast amount of knowledge, with the feedforward layers of the transformer architecture acting as a key-value memory store of knowledge @xcite . this section details experiments conducted to understand the impact of dpr-style training from a model-knowledge perspective. linear probing, a method to characterize model features, involves training a linear classifier on the internal activations of a frozen network to execute a simple task @xcite . this reveals the mutual information shared between the model's primary training task and the probing task @xcite . a high degree of probe accuracy indicates that the model's features possess sufficient information to accomplish the probing task. to evaluate whether dpr training improved bert's discriminative features, linear probing was employed on both pre-trained and dpr-trained bert. a probe g ln (f lq , f ltp , f lhn1 , f lhn2 , . . . ) was trained to classify which passage (between two, three, four, or five passages) is most relevant to a query. the natural questions dataset was used in this experiment, which provides labels for which passages are hard-negatives. the probe g ln at layer l received bert features for the query, true positive passage, and n hard negative passages. f lq represents the features at layer l for the query, f ltp represents the features for the true positive paragraph at layer l, and f lhnn represents the features for the nth hard negative passage at the same layer. a distinct probe g ln was trained for each layer of bert to examine how performance fluctuates across layers and with different numbers of hard-negative passages, thereby assessing how performance is impacted as the task's difficulty increases. the difference between a true positive passage and a hard-negative passage is usually the presence of 1-2 key distinct facts in the passage. the ability to discriminate between 2-5 of these passages indicates that the model likely has enough knowledge to know which facts are relevant to the query. this awareness is likely driven by the model's knowledge of the subject (as discussed in later sections of this paper). rather than testing overall retrieval ability, this experiment aims to find how aware/knowledgeable pre-trained bert's features are compared to dpr-trained bert when the difference of knowing or not knowing 1-2 facts can impact the final matching prediction. ## knowledge decentralization in dpr-trained models the next perspective examined neuron activation patterns for the pre-trained and dpr-trained models. the knowledge attribution method from @xcite was employed which was inspired by the pruning literature @xcite . our analysis targeted linear layers, as this is where the model stores knowledge according to prior research @xcite . to calculate an individual neuron's contribution to the output, we varied its weight w (l) the riemann approximation was used due to the intractability of calculating a continuous integral. following @xcite , a threshold of 0.1 * max(attr) was applied to identify a coarse set of knowledge neurons foot_0 . in contrast to @xcite , the coarse set of knowledge neurons was not refined to a fine set of knowledge neurons, as our interest is in the broader activation patterns. when the model is processing inputs, both "truepositive" and "false-positive" knowledge neurons are activated indiscriminately with the activation strength corresponding to their attribution scores. the primary interest lies in how dpr training influences these activation patterns, rather than the role of specific neurons. figure 1 illustrates the impact of dpr training on bert's neuron activations, charting the attribution score of every neuron across both the intermediate and output linear layers within each transformer block for the query model foot_1 . dpr-trained answer in top-1? # strongly activated neurons title of top-5 retrieval query pretrained bert dpr-bert pretrained bert dpr-bert pre-trained bert dpr bert where is the most distortion on a robinson projection ✗ ✗ 220 1323 circle of latitude, scale-invariant feature transform, line moiré, theil-sen estimator, pole splitting robinson projection, robinson projection, arthur h. robinson, robinson projection, arthur h. robinson who is the chief legal advisor to the government ✗ ✗ 65 831 jimly asshiddiqie, judicial system of iran, comptroller general of the state administration, jimly asshiddiqie, law of kosovo attorney general of india, attorney general, attorney general of india, k. k. venugopal, attorney general of india what type of government does kenya have 2018 ✓ ✗ 74 287 government of kenya, abundant nigeria renewal party, 2007-08 kenyan crisis, independent electoral and boundaries commission, kingdom of kongo government of kenya, politics of kenya, government of kenya, government of kenya, government of kenya are pure metals made of atoms or ions ✓ ✗ 69 1268 alloy, common attributes, metal, resonance ionization, alloy properties of metals, metalloids and nonmetals, properties of metals, metalloids and nonmetals, solid, metal, metal who is the bad guy in lord of the rings ✗ ✓ 100 533 millennium earl, the sword of shannara, eye of ra, the enchanted apples of oz, ys i & ii saruman, saruman, sauron, morgoth, legolas when were manatees put on the endangered list ✗ ✓ 42 1522 ivory trade, namib desert horse, endangered species act of 1973, iriomote cat, bile bear manatee conservation, endangered species act of 1973, endangered species act of 1973, manatee conservation, endangered species act of 1973 when did wesley leave last of the summer wine ✓ ✓ 38 1024 naif (band), aiden, queensrÿche, josef brown, matthew stocke gordon wharmby, gordon wharmby, brian wilde, cory monteith, last of the summer wine when did mozart compose his first piece of music ✓ ✓ 74 364 wolfgang amadeus mozart, der messias, life of franz liszt, die entführung aus dem serail, quattro versioni originali della ritirata notturna di madrid¨w olfgang amadeus mozart, wolfgang amadeus mozart, leopold mozart, wolfgang amadeus mozart, wolfgang amadeus mozart bert has more activated neurons in the intermediate layer of each block. the output layer, on the other hand, maintains a consistent number of activations at each transformer block compared to pre-trained bert, and in the earlier layers dprtrained bert activates fewer neurons in the output layers. previous studies have conceptualized intermediate layers as "keys" and the output layer as the "value" @xcite . this suggests that dpr training expands the set of "keys" available to access a given volume of semantic knowledge while decreasing the accessible volume of syntactic knowledge, embodying a decentralization strategy for semantic knowledge. rather than relying on a single, highly precise key to unlock some knowledge, dpr allows for the use of multiple, somewhat less precise keys. this underscores dpr training's primary goal: to modify the model's method of knowledge access without altering the stored knowledge itself. these multiple pathways enable morphologically distinct but semantically related text to trigger the same knowledge or collections of facts, thus making retrieval possible. ## knowledge editing experiments if dpr rearranges knowledge in the pre-trained bert model, are these facts discoverable in dpr-bert? we employed model editing techniques to remove facts from pre-trained bert to investigate this. currently, models struggle to learn and update the knowledge within them. model editing is a class of techniques that minimizes pre-trained models to add or remove knowledge without disturbing the overall network. the types of methods generally used for model editing can be broadly categorized as fine-tuning with regularization, direct model editing, meta-learning, and architectural @xcite . owing to the emerging state of this subfield and the variability in results, we employed various model editing techniques. in selecting techniques, we prioritized locality-preserving methods that minimally altered the model. this was to facilitate clearer attributions of our findings to dpr training rather than to potential architectural modifications. this led to transformerpatch, malmen, and mend being chosen to perform the model editing @xcite @xcite . transformerpatch is an architecturemodifying technique that introduces a single parameter to the last layer for each fact added. the experiments involving transformerpatch used a learning rate of 3e-5. malmen and mend, on the other hand, are meta-learning techniques. they utilize hypernetworks, a type of network that learns to generate parameters or parameter shifts for another model @xcite . these hypernetworks were trained using a learning rate of 1e-6 and to only modify the last three layers of the bert network. ## conclusion this paper analyzes the effect of dpr fine-tuning using different analysis methods. through these analyses, we find that: if dpr does not add facts, then we should be able to remove relevant facts that pre-trained bert knows and not find them again in dpr-trained bert. section 4 indicates that this statement is true, suggesting from another perspective, that dpr is dependent on the knowledge already contained in the model to produce satisfactory retrievals. if a fact is removed, it will likely not be added back during dpr training. thus the existence or absence of necessary facts or webs of knowledge within a model can aid or hamper its ability to retrieve information. in the most fundamental sense, dpr achieves its namesake function-it retrieves, locating and returning relevant context to the user given a query. yet, as our evidence suggests, dpr models appear constrained to retrieving information based on the knowledge that preexists within their parameters, either innately or through augmentation. this operational boundary delineates a significant caveat: facts must already be encoded within the model for useful context to be accessible by retrieval. absent these facts or their associative networks, retrieval seems to falter. thus, if retrieval is understood as the capacity to recall or recognize knowledge already familiar to the model or on the periphery of what is familiar, then indeed, dpr models ful-fill this criterion. however, if we extend our definition of retrieval to also encompass the ability to navigate and elucidate concepts previously unknown or unencountered by the model-a capacity akin to how humans research and retrieve information-our findings imply that the current dpr model fall short of this mark. our findings suggest several areas of focus for future work including (1) accelerate knowledge representation decentralization with new unsupervised training methods (2) develop new methods to inject facts in a decentralized manner into the network (3) optimize retrieval methods that operate with uncertainty, and (4) map the model's internal knowledge directly to the set of best documents to retrieve. current work in optimizing the inverse cloze pre-training task and various data augmentation methods such as @xcite begin to address (1) by increasing the amount of knowledge that the model is exposed to during fine-tuning and thus the amount that can be decentralized. with the knowledge of the purpose of dpr-training more targeted methods can be developed. (3) requires more detailed analysis to determine how a model processes a query when it is missing key knowledge needed for retrieval. being aware of when a model is uncertain in its retrieval is crucial. the analysis should reveal methods to more robustly and gracefully handle increased levels of uncertainty. one direction to better leverage a model's knowledge as suggested in (4) is shown in @xcite @xcite @xcite .