Self-Prompting Large Language Models for Zero-Shot Open-Domain QA (2024)

Abstract: Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing specific background documents. This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models.While recent Large Language Models (LLMs) like GPT-3 have demonstrated their effectiveness in zero-shot ODQA using direct prompting methods, these methods still fall short of fully harnessing the potential of LLMs when implicitly invoked.In this paper, we propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of LLMs and their strong instruction understanding abilities. Concretely, we prompt LLMs step by step to generate multiple pseudo QA pairs with background passages and explanations entirely from scratch.These generated elements are then utilized for in-context learning. Experimental results show that our method significantly surpasses previous state-of-the-art zero-shot methods on three widely-used ODQA datasets and even achieves comparable performance with various customized fine-tuned models on full training data. Our code is available at https://github.com/lockon-n/self-prompting.

## introduction open-domain question answering (odqa) is a longstanding task in natural language processing that aims to answer questions about world knowledge without explicitly providing specific background documents @xcite @xcite . the most common approach for odqa is to train a retriever-reader pipeline @xcite : (i) first retrieving the most related documents of the question, (ii) then applying the reader to extract or generate the final answer conditioned on these documents @xcite @xcite . despite the decent performance, they rely on large amount of training data and a large external knowledge corpus, so it is hard to expand these methods to zero-shot odqa where no training data is available. with the emergence of large langauge models (llms) such as gpt-3 @xcite , opt @xcite , palm @xcite , instructgpt @xcite , researchers start to leverage them for zero-shot odqa tasks. these llms are capable of generating correct answers without any training data and external corpus with direct prompts (like q: {question} a:). some recent works go further to induce the instruction understanding abilities and use the abundant knowledge in their parameters by asking llms to generate rationales called chainof-thought @xcite or related background information @xcite before answering. however, these naive prompting methods for zero-shot odqa still fail to compete with customized fine-tuned models @xcite , since they fail to take full advantage of the powerfulness of llms. in this paper, we focus on zero-shot odqa with no training data and external corpus. based on the characteristics of this task, we propose self-prompting the llm to explicitly activate different capabilities of llms and combine these capabilities to improve performance. in general, we ask llms to automatically generate pseudo qa pairs for in-context learning instead of predicting the answers directly as in naive prompting methods. self-prompting consists of two stages: preparation and inference. in the preparation stage, the llm is required to generate a pseudo qa dataset in three steps: (i) write a short wikipedia-style passage, extract named entities in this passage as answers; we evaluate our approach on three odqa benchmarks: webq @xcite , nq @xcite , and triviaqa @xcite . experimental results show that our method significantly surpasses the direct prompting baselines and the previous state-of-the-art (sota) zero-shot method genread @xcite . our method even shows comparable performance with strong few-shot methods. we also conduct extensive analysis on the effects of different generated components, like the formats of the prompted sequence for in-context learning, ways of demonstration selection from the pseudo dataset, and the quality of generated qas. in general, our contributions can be summarized as follows: llm and in-context learning generally, llms refer to the pre-trained models with tens or hundreds of billions of parameters, such as gpt-3, flan, palm, opt, and instructgpt @xcite @xcite @xcite . these models are trained with large-scale unsupervised learning and are able to perform nlp tasks by converting inputs into natural language queries without further training. the cost of fine-tuning these models is extremely huge, so the usual way of using them is in-context learning, which is to put some input-output pairs as demonstrations in front of the test sample. some previous works have investigated the calibration @xcite , example selection @xcite and ordering @xcite of in-context learning, while to the best of our knowledge, we are the first to generate a large number of demonstrations used in in-context learning completely by the llm itself from scratch. ## approach this section presents the details of self-prompting, which has two stages: preparation and inference. in the first stage, we require the llm to automatically build a pseudo odqa dataset by prompting it to generate qa pairs with context passages and explanations. in the second stage, we dynamically select examples in the pool through a clustering-based retrieval method as in-context demonstrations to help understand and answer the given question. an overall framework is shown in figure 1 . ## conclusion in this paper, we propose self-prompting llms for zero-shot open-domain question answering. our method induces the llm to generate pseudo qa pairs with matching passages and explanations and use them for in-context learning. it successfully stimulates the potential of the llm in odqa by explicitly activating various language understanding abilities and eliciting knowledge in its parameters. with no training data and external corpus, self-prompting surpasses previous sota significantly and performs on par with several strong retrieval-augmented fine-tuned models and fewshot prompting methods. it is generally effective across llms under different sizes and is able to provide accurate and helpful explanations alongside the answers, making it a more reliable and interpretable framework for turstworthy ai systems.