Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks (2023)

Abstract: Retrieval-augmented methods have received increasing attention to support downstream tasks by leveraging useful information from external resources. Recent studies mainly focus on exploring retrieval to solve knowledge-intensive (KI) tasks. However, the potential of retrieval for most non-knowledge-intensive (NKI) tasks remains under-explored. There are two main challenges to leveraging retrieval-augmented methods for NKI tasks: 1) the demand for diverse relevance score functions and 2) the dilemma between training cost and task performance. To address these challenges, we propose a two-stage framework for NKI tasks, named PGRA. In the first stage, we adopt a task-agnostic retriever to build a shared static index and select candidate evidence efficiently. In the second stage, we design a prompt-guided reranker to rerank the nearest evidence according to task-specific relevance for the reader. Experimental results show that PGRA outperforms other state-of-the-art retrieval-augmented methods. Our analyses further investigate the influence factors to model performance and demonstrate the generality of PGRA. The code and model will be released for further research.

## introduction retrieval-augmented methods aim at enhancing dense models with non-parametric indices to better leverage external knowledge @xcite @xcite . by decoupling knowledge storage from model parameters, retrieval-augmented methods can achieve comparable or better performance than large-scale pre-trained models with orders of magnitude less parameters on tasks such as language modeling @xcite @xcite and question answering @xcite @xcite . moreover, as external knowledge is stored in the non-parametric index, knowledge can be updated simply by replacing the index without further training @xcite . therefore, retrieval-augmented methods have attracted increasing interest in recent years and achieved promising results in various natural language processing tasks @xcite @xcite . despite their success, retrieval-augmented methods for the majority of non-knowledge-intensive (nki) tasks remain under-explored. following @xcite , we define tasks that "humans could not reasonably be expected to perform without access to an external knowledge source" as knowledge-intensive (ki) tasks and the others as nki tasks. previous studies @xcite @xcite have extensively explored the potential of retrievalaugmented methods for various ki tasks. as for nki tasks, most efforts are devoted to language modeling @xcite ), text generation @xcite , and machine translation @xcite , although there is a wide range of nki tasks, such as sentiment analysis @xcite ), text classification @xcite and linguistic acceptability @xcite . therefore, we ask this question: can retrieval-augmented methods assist on a wider range of nki tasks? however, leveraging retrieval-augmented methods for more types of nki tasks faces two major challenges. on the one hand, there is a demand for diverse relevance score functions. to retrieve the most desirable evidence from the index, proper relevance score functions are needed. although the relevance score functions suitable for predicting the next token distribution are well-studied in works on language modeling, text generation, and ma-task-agnostic retriever data reader q: this is the biggest insult to tmnt ever. t1: teenage mutant ninja turtles is an american media franchise. t2: danger of the ooze is still struggles to completely find its footing. t3: the tmnt alphabet insult game! t4: tmnt: danger of the ooze does a few things right, but it still feels incredibly generic. template: is this review positive or negative? the answer is: template t1 prompt-guided reranker answer: negative top-d (@xmath0 figure 1 : the framework of our proposed prompt-guided retrieval augmentation (pgra) method. we first retrieve candidates through a task-agnostic retriever (section 2.1), then use a task-specific prompt and pre-trained language model (plm) to rerank the candidates (section 2.2). we send the top results to the reader to make predictions. (section 2.3). to address these challenges, we propose a twostage framework, entitled pgra, to better retrieve task-specific resources for nki tasks. the overall framework is shown in figure 1 . in the first stage, we use a task-agnostic retriever to recall candidate evidence, which builds a shared static index for all tasks. in the second stage, we adopt prompt-guided pretrained language models (plms; @xcite as a reranker to rerank the candidates according to the task-specific relevance score functions. finally, we feed the reranked top evidence to the reader to generate answers. by leveraging textual prompts, our framework can satisfy the demand for diverse relevance score functions. as both the retriever and the reranker are training-free, the expensive computational cost of periodical index update in training is avoided. at the same time, experimental results justify the effectiveness of our framework on various datasets. therefore, we successfully break the dilemma between training cost and task performance. our main contributions are three-fold: ## methods in this section, we introduce our proposed prompt-guided retrieval augmentation (pgra) method, as shown in figure 1 . our proposed method mainly has three components: (i) a task-agnostic retriever using a shared retriever to build static indexes to select top-k candidate evidence from large-scale external resources; (ii) a prompt-guided reranker adopting plms to measure task-specific relevance for reranking candidate evidence; (iii) a reader taking the final top-d (@xmath1. ## related work retrieval-augmented methods are widely used for knowledge-intensive tasks such as question answering @xcite ; izac- ## conclusion in this paper, considering the demand for diverse relevance score functions to solve wider nki tasks, we propose a two-stage method pgra. in the first stage, we use a task-agnostic retriever for building shared static indexes to select candidate evidence. in the second stage, we design a prompt-guided reranker to rerank candidates with task-specific relevance for the reader. extensive experimental results show that our proposed method pgra can overall outperform previous state-of-the-art retrieval-augmented methods. furthermore, we explore the influence of label consistency between input and retrieved evidence from the prompt-guided reranker and demonstrate the generality of our pgra on both evidence granularities and types of retrievers. in the future, we will consider ways to improve the pseudo-label consistency to enhance model performances according to our analyses.