Topic- DPR : Topic-based Prompts for Dense Passage Retrieval (2023)

Abstract: Prompt-based learning’s efficacy across numerous natural language processing tasks has led to its integration into dense passage retrieval. Prior research has mainly focused on enhancing the semantic understanding of pre-trained language models by optimizing a single vector as a continuous prompt. This approach, however, leads to a semantic space collapse; identical semantic information seeps into all representations, causing their distributions to converge in a restricted region. This hinders differentiation between relevant and irrelevant passages during dense retrieval. To tackle this issue, we present Topic-DPR, a dense passage retrieval model that uses topic-based prompts. Unlike the single prompt method, multiple topic-based prompts are established over a probabilistic simplex and optimized simultaneously through contrastive learning. This encourages representations to align with their topic distributions, improving space uniformity. Furthermore, we introduce a novel positive and negative sampling strategy, leveraging semi-structured data to boost dense retrieval efficiency. Experimental results from two datasets affirm that our method surpasses previous state-of-the-art retrieval techniques.

## introduction dense passage retrieval (dpr), due to its efficacy and efficiency, has gained significant attention recently @xcite . dpr encapsulates semantic information of queries and passages within a low-dimensional embedding space and measures relevance using cosine distance. prompt-based learning is an effective emerging technique for multiple natural language processing tasks @xcite @xcite . this technique uses a task-specific prompt as input to augment the performance of pretrained language models (plms). prompts, typically discrete text templates with task-specific information, need explicit definitions. to circumvent local optima during tuning, researchers @xcite suggested deep prompt tuning that trains a single vector as a continuous prompt. this approach demonstrated effectiveness and flexibility in text-generation tasks. inspired by deep prompt tuning, recent research @xcite has integrated continuous prompts into retrieval tasks. by adding task-specific semantic information as input, these prompts improve plms' knowledge utilization and guide plms to produce more relevant text representations. consequently, relevant passage representations are likely closer to the query, thus securing higher rankings. however, past research has not fully addressed the limitations of a single prompt when dealing with the diverse semantics of a comprehensive dataset. the engagement of all passages with a singular prompt induces a uniform semantic shift. imagine using a single prompt like "explain the main concept of @xcite ," where @xcite is the actual query. while this prompt may effectively steer the pre-trained language model (plm) to generate representations capturing aspects of each discipline, it may overlook the diverse semantics and terminologies across fields. consequently, it could have difficulty differentiating articles from distinct disciplines. a single prompt might fall short of capturing the nuances within each discipline, like subtopics or specialized areas. employing one prompt for all disciplines may result in a uniform semantic shift and a convergence of passage representations in a restricted region of the embedding space, as depicted in figure 1 . this semantic space collapse @xcite @xcite can blur the distinction between relevant and irrelevant passages, potentially masking irrelevant passages amidst relevant ones. therefore, prompt generation is pivotal for this semantically nuanced task. further analysis is conducted in section 5. in this paper, we explore the use of multiple continuous prompts to address the anisotropic issue of deep prompt tuning in dense passage retrieval, a persisting challenge. challenge 1: the effective generation of multiple continuous prompts based on the corpus. a simple approach is partitioning the dataset into subsets via topic modeling, each sharing a common topic-based prompt. this strategy allows the distribution of latent topics across a probabilistic simplex and unsupervised extraction of semantics @xcite , enabling the definition and initialization of distinct, interpretable topic-based prompts. challenge 2: the integration of topic-based prompts into the pre-trained language models (plms). although our topic-based prompts are defined on a probabilistic simplex using topic modeling, ensuring topical independence, constructing such a simplex and learning topical knowledge within the plms' embedding space presents a challenge due to inherent model differences. as a result, we make the topic-based prompts trainable and adopt contrastive learning @xcite for optimizing topical relationships. to tackle these challenges, we introduce a novel framework, topic-dpr, that efficiently incorporates topic-based prompts into dense passage retrieval. instead of artificially inflating the number of prompts, we aim to define a prompt set reflecting the dataset's diverse semantics through a data-driven approach. we propose a unique prompt generation method that utilizes topic modeling to establish the number and initial values of the prompts, which we term topic-based prompts. these prompts are defined within a probabilistic simplex space @xcite , initialized using a topic model such as hierarchical latent dirichlet allocation (hlda) @xcite . moreover, we propose a loss function based on contrastive learning to preserve the topic-topic relationships of these prompts and align their topic distributions within the simplex. the impact of topic-based prompts serves as a pre-guidance for the plms, directing representations towards diverse sub-topic spaces. for dense retrieval, we consider query similarities and design a tailored loss function to capture query-query relationships. we use contrastive learning to maintain query-passage relationships, maximize the similarity between queries and relevant passages, and minimize the similarity between irrelevant pairs. considering the semi-structured nature of the datasets, we also introduce an in-batch sampling strategy based on multi-category information, providing high-quality positive and negative samples for each query during fine-tuning. the efficacy of our methods is confirmed through comprehensive experiments, emphasizing the role of topic-based prompts within the topic-dpr framework. the key contributions are: ## conclusion in this paper, we examine the limitations of using a single task-specific prompt for dense passage retrieval. to address the anisotropic issue in the embedding space, we introduce multiple novel topic-based prompts derived from the corpus's semantic diversity to improve the uniformity of the space. this approach proves highly effective in identifying and ranking relevant passages during retrieval. we posit that the strategy of generating data-specific continuous prompts may have broader applications in nlp, as these prompts encourage plms to represent more diverse semantics.