DPTDR : Deep Prompt Tuning for Dense Passage Retrieval (2022)

Abstract: Deep prompt tuning (DPT) has gained great success in most natural language processing (NLP) tasks. However, it is not well-investigated in dense retrieval where fine-tuning (FT) still dominates. When deploying multiple retrieval tasks using the same backbone model (e.g., RoBERTa), FT-based methods are unfriendly in terms of deployment cost: each new retrieval model needs to repeatedly deploy the backbone model without reuse. To reduce the deployment cost in such a scenario, this work investigates applying DPT in dense retrieval. The challenge is that directly applying DPT in dense retrieval largely underperforms FT methods. To compensate for the performance drop, we propose two model-agnostic and task-agnostic strategies for DPT-based retrievers, namely retrieval-oriented intermediate pretraining and unified negative mining, as a general approach that could be compatible with any pre-trained language model and retrieval task. The experimental results show that the proposed method (called DPTDR) outperforms previous state-of-the-art models on both MS-MARCO and Natural Questions. We also conduct ablation studies to examine the effectiveness of each strategy in DPTDR. We believe this work facilitates the industry, as it saves enormous efforts and costs of deployment and increases the utility of computing resources. Our code is available at https://github.com/tangzhy/DPTDR .

## introduction fine-tuning (ft) has been a de facto approach for effective dense passage retrieval @xcite based on pre-trained language models (plm). however, ft is unfriendly for industrial deployment in multi-task scenarios. imaging for cloud service providers or infrastructure teams of search companies, each retrieval model (w.r.t., an individual task) necessarily redeploys a backbone model since the weights of 1 our code is available at https://github.com/ tangzhy/dptdr the backbone model in each task are fine-tuned and therefore slightly different. that dramatically increases hardware costs and inefficiency. recently, prompt tuning (pt) @xcite ) is a lightweight alternative to ft, which does not need storing a full copy of the backbone model for each task. one variant of pt, namely deep prompt tuning (dpt; @xcite , exhibits comparable performances with ft in various nlp tasks. dpt enjoys parameterefficient @xcite characteristics, of which the resulting prompts are light-weighted and can be easily passed to an online plm service, thus overcoming the above challenge of ft. this paper asks: whether can we replace ft by dpt with comparable performance to sota ft methods in dense passage retrieval? with comparable performance, dpt is much more friendly in deployment than ft. dpt usually freezes weights in the backbone models and alternatively trains deep prompts inserted; the latter has much fewer parameters than the former. however, freezing most weights in dpt hinders its adaptability and therefore possibly harms performance. experimental results in sec. 4.2.2 also demonstrate directly applying dpt in dense retrieval largely underperforms ft methods. to make dpt comparable to ft in dense retrieval, a natural solution is retrieval-oriented intermediate pretraining (rip), which warms up the text representation via contrastive learning. though it is not a novel idea @xcite @xcite , there exist two different pretraining ways tailored for dpt-based retrievers. one is to pre-train deep prompts while freezing the plm backbone and use the pre-trained prompts to initialize a dpt retriever. the other is to pre-train a plm directly and initialize a dpt retriever using the pre-trained plm; in contrast to prior works @xcite , we intend to allow any plm easily pre-trained for dpt so that users may employ their own plms, and thus we deliberately remove the workload to modify any model structures. surprisingly, empirical findings in sec. 4.4 show that this choice yields better performance than carefully modified plms @xcite . furthermore, we propose a unified negative mining (unm) to merge retrieved negatives from many retrievers including bm25 and dense retrievers, in order to provide diverse and hard negatives for dpt training. by incorporating rip and unm, we implement a deep prompt tuning method in dense retrieval tasks, called dptdr. the experimental results show that dptdr outperforms previous state-ofthe-art models on both ms-marco and natural questions. we also conduct extensive experiments and find that: i) when combined with rip and unm, dpt is able to obtain comparable performance with ft in dense retrieval and exhibits insensitivity to prompt length, and ii) both rip and unm are effective in improving the performance. the contributions of this paper can be summarized as follows: ## methodology in this section, we first formalize the application of dpt in dense retrieval. we then describe the two strategies of rip and unm for dpt-based retrievers. figure 1 : the framework of dptdr. we first perform rip which results in a plm (the blue blocks) that can be used as the backbone for dpt training and deployed once as online plm services. then we train deep prompts (i.e., dpt) for different retrieval tasks such as webqa, wikiqa, and medicalqa (the pink blocks), during which we may employ unm to improve performances. for inference, we can send tokenized input, together with trained prompts of their corresponding task, to online plm services to get dense vectors. ## conclusion in this paper, we investigate applying dpt in dense passage retrieval. to mitigate the performance drop of a vanilla dpt, we also propose two strategies, namely rip and unm, to enhance dpt and match the performance of ft. experiments show that dptdr outperforms previous state-of-the-art models on both ms-marco and natural questions and demonstrated the effectiveness of the above strategies. we believe this work facilitates the industry, as it saves enormous efforts and costs of deployment and increases the utility of computing resources. in future work, we will explore scaling up the model size to further improve dptdr.