Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers (2023)

Abstract: Prompt tuning attempts to update few task-specific parameters in pre-trained models. It has achieved comparable performance to fine-tuning of the full parameter set on both language understanding and generation tasks. In this work, we study the problem of prompt tuning for neural text retrievers. We introduce parameter-efficient prompt tuning for text retrieval across in-domain, cross-domain, and cross-topic settings. Through an extensive analysis, we show that the strategy can mitigate the two issues—parameter-inefficiency and weak generalizability—faced by fine-tuning based retrieval methods. Notably, it can significantly improve the out-of-domain zero-shot generalization of the retrieval models. By updating only 0.1% of the model parameters, the prompt tuning strategy can help retrieval models achieve better generalization performance than traditional methods in which all parameters are updated. Finally, to facilitate research on retrievers’ cross-topic generalizability, we curate and release an academic retrieval dataset with 18K query-results pairs in 87 topics, making it the largest topic-specific one to date.

## introduction seeking for relevant texts has been a fundamental problem for a broad range of natural language processing (nlp) applications such as open-domain question answering @xcite , retrievalaugmented language modeling @xcite , and fact verification @xcite . its recent progress has been dominantly favored by the neural approaches @xcite , especially the large-scale pre-trained language models with ever-growing parameters. for example, a recent study attempts to leverage models up to 10 billion parameters @xcite ## related work neural text retrieval. text retrievers traditionally rely on sparse lexical-based inverted index to rank candidate documents containing query terms (e.g., tf-idf and bm25). they benefit from the simplicity but often suffer from the lexical gap @xcite . recently, neural text retrievers, including dense retrievers @xcite @xcite , late-interaction models @xcite , and hybrid or re-ranking models @xcite , becomes popular as they can capture the semanticlevel query-document similarity thanks to the advance of pre-trained language models @xcite . generalization in text retrieval. the weaker generalizability of neural retrievers compared to conventional lexical ones has recently arouse concerns in the community @xcite , and it results in beir, a heterogeneous cross-domain generalization benchmark @xcite . while recent works notice and employ ideas like bigger pre-trained models @xcite or unsupervised pre-training on large corpus @xcite to improve scores on beir, few of them focus on studying better transferring strategies based on existing architectures and datasets for out-of-domain generalization. parameter-efficient (pe) learning. sizes of pretrained language models are soaring up @xcite , causing great challenges to traditional task transfer based on full-parameter finetuning. a recent focus has been on the emerged pe transfer learning, including prompt tuning @xcite @xcite , adapters @xcite , and hybrid methods @xcite . they employ very few tuning parameters to achieve finetuning comparable transfer performance. despite abundant research made on problems like language understanding @xcite and generation (li and liang, 2021), how it will impact retrieval remains under-explored. ## challenges in neural text retrieval the neural text retriever, which leverages pretrained language models, e.g., bert @xcite and roberta @xcite , as the backbone, has significantly mitigated the lexical gap @xcite in text retrieval and become a standard component for many nlp applications @xcite @xcite . it consists of several different categories and in this work we focus on the following two dominant ones. ## parameter-efficient transfer learning we introduce the parameter-efficient transfer learning (pe learning) framework and notable techniques. different from fine-tuning @xcite , which updates the full parameters of pretrained models for each target task, pe learning aims to achieve comparable performance to finetuning by tuning only a small portion of parameters per task @xcite @xcite . ## cross-domain and cross-topic generalizability in this section, we examine the zero-shot generalizability of fine-tuning and pe learning. we take p-tuning v2 @xcite as an representative for pe methods, which has the highest average indomain accuracy . particularly, as previous work seldom looks into the cross-topic generalization, we introduce oag-qa, the largest fine-grained cross-topic retrieval dataset to date. on crossdomain evaluation, we adopt the well-acknowledge beir @xcite benchmark. ## an understanding of the generalization how does pe learning help neural text retrievers to generalize well? while it might be attributed to pe learning's flatter loss minimum or alleviated catastrophic forgetting, in this work we investigate other quantifiable reasons, the confidence calibration and query-length robustness. ## conclusion we propose to leverage pe prompt tuning for neural text retrieval, which is proved for the first time in this problem for comparable performance to full-parameter fine-tuning. furthermore, pe approaches like p-tuning v2 improve cross-domain and cross-topic generalization, which fundamentally comes from improved confidence calibration and query length robustness as we show. finally, we construct and release the largest fine-grained topic-specific academic retrieval dataset oag-qa, which contains 87 different domains and 17,948 query-paper pairs, to support future research.