QP aug: Question and Passage Augmentation for Open-Domain Question Answering of LLM s (2024)

Abstract: Retrieval-augmented generation (RAG) has received much attention for Open-domain question-answering (ODQA) tasks as a means to compensate for the parametric knowledge of large language models (LLMs). While previous approaches focused on processing retrieved passages to remove irrelevant context, they still rely heavily on the quality of retrieved passages which can degrade if the question is ambiguous or complex. In this paper, we propose a simple yet efficient method called question and passage augmentation (QPaug) via LLMs for open-domain QA. QPaug first decomposes the original questions into multiple-step sub-questions. By augmenting the original question with detailed sub-questions and planning, we are able to make the query more specific on what needs to be retrieved, improving the retrieval performance. In addition, to compensate for the case where the retrieved passages contain distracting information or divided opinions, we augment the retrieved passages with self-generated passages by LLMs to guide the answer extraction. Experimental results show that QPaug outperforms the previous state-of-the-art and achieves significant performance gain over existing RAG methods.

## introduction large language models (llms) have shown remarkable in-context learning capability @xcite for various real-world applications such as assistant chatbot @xcite , robot planning @xcite , search ranking @xcite , and code generation @xcite . however, the knowledge of llms is limited to the pre-training corpus, making it difficult * corresonding author to provide answers to questions on up-to-date information. to overcome such limitations via external knowledge sources, recent works have focused on the retrieval-augmented-generation (rag) @xcite . the rag systems consist of retrievers which search and retrieve related information from knowledge sources and readers which generate responses based on the retrieved information. the rag approach has been proven effective in leveraging external knowledge to complement the parametric knowledge of llms. open-domain question answering (odqa) @xcite @xcite ) is one of the nlp tasks most relevant to rag systems. in the odqa, retrievers search for relevant passages from questions, and readers answer the questions based on retrieved contexts. research efforts have been put into enhancing retrievers @xcite and readers @xcite . for example, raptor @xcite is an advanced retriever which recursively captures multiple levels of details of a text using a tree. however, even advanced retrievers may fetch poor passages when the question is ambiguous or complex, requiring question clarification @xcite @xcite or multi-step retrieval @xcite @xcite , rendering the retrieval inaccurate and inefficient. thus, for tough questions, it is unclear whether the retrieved passages are of high quality, i.e., they contain context relevant to the question, which makes the design of readers challenging. the problem can be alleviated if the parametric knowledge of llms can be tuned to extract answers given the relevant retrieved passages @xcite . however, fine-tuning is often infeasible due to the sheer scale of llms, or some llms are essentially black-box apis @xcite . recently, @xcite have proposed to process retrieved passages via llm prompting without fine-tuning, but with internet retrieval and reranking @xcite , or with summarization and verification @xcite . however, these approaches are still limited in that they heavily rely on the quality of contexts provided by retrieved passages. in this paper, we deal with question complexity and complement the quality of retrieval by incontext learning. the goal is to harmoniously combine the parametric and non-parametric knowledge of llms through prompting. we propose qpaug (pronounced cue-pug) which stands for question and passage augmentation. firstly, the question augmentation is based on the hypothesis that the llms can decompose complex questions into multiple easier sub-questions @xcite . the sub-questions contain fine-grained information and planning as to what knowledge should be retrieved from external sources. the question for retrieval is composed by augmenting the original question with the sub-questions. secondly, the passage augmentation is based on the hypothesis that the llms' parametric knowledge is the most pragmatic alternative source to non-parametric (retrieved) knowledge to deal with the degraded quality of retrieval. a self-generated passage is composed with respect to the augmented question, leveraging extensive knowledge of llms. also, self-generation can prevent irrelevant retrieved passages from overriding the llms' factual knowledge @xcite . we augment the retrieved passages with the self-generated passages. experiments show that the question and passage augmentation through llms complement the retrieved generation well, achieving state-ofthe-art performance on odqa benchmark datasets. our contributions are summarized as follows. (i) we propose a simple yet effective framework for the llm-based question and passage augmentation method (qpaug) based on the prompting of llms. (ii) qpaug successfully decomposes and augments questions improving the performance of retrieval. in addition, the self-generated passage by qpaug, when combined with retrieved passages, strengthens the factual knowledge for the reader, achieving large performance gains. (iii) extensive ablation studies show that qpaug can be integrated with various llms and retrievers, exhibiting excellent performances on several odqa benchmarks. ## ablation study we conduct ablation study on two main components, i.e., question augmentation and passage generation, denoted by qaug and pgen in the following subsections. ## conclusion in this paper, we propose a simple yet effective question and passage augmentation (qpaug) method via llms. qpaug harmoniously combines parametric and non-parametric knowledge of llms by leveraging the massive knowledge of llms for designing queries for retrieval and for guiding answer extraction from passages. experiments show that both proprietary and opensource llms successfully decompose questions into multiple sub-questions by the proposed question augmentation, which significantly improved the performances of passage retrieval. in addition,