Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generation (2024)

Abstract: Despite the significant progress of large language models (LLMs) in various tasks, they often produce factual errors due to their limited internal knowledge. Retrieval-Augmented Generation (RAG), which enhances LLMs with external knowledge sources, offers a promising solution. However, these methods can be misled by irrelevant paragraphs in retrieved documents. Due to the inherent uncertainty in LLM generation, inputting the entire document may introduce off-topic information, causing the model to deviate from the central topic and affecting the relevance of the generated content. To address these issues, we propose the Retrieve-Plan-Generation (RPG) framework. RPG generates plan tokens to guide subsequent generation in the plan stage. In the answer stage, the model selects relevant fine-grained paragraphs based on the plan and uses them for further answer generation. This plan-answer process is repeated iteratively until completion, enhancing generation relevance by focusing on specific topics. To implement this framework efficiently, we utilize a simple but effective multi-task prompt-tuning method, enabling the existing LLMs to handle both planning and answering. We comprehensively compare RPG with baselines across 5 knowledge-intensive generation tasks, demonstrating the effectiveness of our approach.

## introduction with the persistent scaling up of training parameters and datasets @xcite , large language models (llms) @xcite @xcite have made remarkable advancements, becoming the cornerstone of many natural language processing (nlp) tasks in recent years. despite improve-figure 1 : the retrieval documents contain off-topic paragraphs (highlighted in yellow), causing potential deviations in rag outputs. by planning first (highlighted in green), selecting relevant fine-grained paragraphs, and then answering, the plan-answer iteration ensures a more consistent and relevant generation. ments in model architecture and the expansion of training data, llms still struggle with factual errors @xcite . to address this issue, the retrieval-augmented generation (rag) system has been introduced @xcite . by retrieving external information and incorporating it into the input, the rag system demonstrates excellent performance in knowledge-intensive tasks. the most common approach in rag involves using the user input as a query for a single-time retrieval @xcite , with llms then generating answers based on the retrieved information. however, documents retrieved for input into the llm are often lengthy, and not all paragraphs may be practically helpful for answering the question. recent research @xcite indicates that off-topic paragraphs can be detrimental to the generation. as figure 1 illustrates, due to the inherent uncertainty in the generation process of llms, inputting the entire retrieved document can lead to those off-topic paragraphs misleading the model, causing a shift in focus and resulting in content that gradually deviates from the main topic. currently, many researchers have acknowledged this issue and have adopted various solutions. some works @xcite determine whether retrieval is necessary before generating an answer and input the retrieved document only when required. self-rag @xcite further introduces reflection tokens to evaluate the quality of retrieved documents, thereby excluding irrelevant documents. despite significant advancements with these methods, their effectiveness diminishes when dealing with longer retrieved texts, particularly those that are generally relevant but contain some irrelevant details. additionally, when the retrieved documents are too lengthy, it becomes challenging for users to verify the correctness of specific details in the generated content. we propose that the susceptibility of llms to irrelevant content stems from a lack of explicit pre-planning in generating subsequent content. as illustrated in figure 1 , if the model continuously plans the next topic at each step and only focuses on highly relevant paragraphs, it can avoid being misled by irrelevant material during lengthy generation processes. to implement this plan-answer process, we introduce the retrieve-plan-generation (rpg) framework. rpg iterates through two stages: the plan stage and the answer stage. in the plan stage, the model generates tokens representing upcoming text topics. during the answer stage, the model selects highly on-topic paragraphs from retrieved documents based on these topics, and uses them to generate targeted answers. this iterative process between planning and answering continues until the generation is complete. unlike traditional full-text input methods, rpg provides detailed control over content generation by focusing on specific topics at each step, ensuring the generation is highly relevant and accurate. additionally, this fine-grained approach makes it easier for users to verify the correctness of answer details, even when dealing with long documents. existing llms struggle to effectively integrate both planning and answering capabilities. since the plan must be incrementally developed during the generation process, relying solely on pre-designed prompts for plan generation is challenging. additionally, prompts need to guide the model in generating both the plan and the answer based on generated context and relevant paragraphs, which imposes high demands on the model's ability to comprehend complex prompts. therefore, we prompt chatgpt to create supervision for plan generation and fine-grained paragraph utilization based on existing datasets @xcite , then train our model end-to-end on this dataset. fully fine-tuning an llm is resource-intensive and often unnecessary. to balance the learning capabilities of the llms with training efficiency, prompt tuning has emerged as a promising method. given that the input and output formats for planning and answering tasks differ, we adopt a multitask prompt tuning approach, training two learnable prompt tokens specifically for plan and answer generation. these two prompt tokens share the same soft prompt. during the training stage, each data is simultaneously used for both planning and answering tasks. to train task-specific prompts, we first transform the soft prompt to the corresponding task mode, and then exclude the impact of other parts during loss computation. empirical results on 5 tasks, including longform, multi-hop, and short-form generation, demonstrate that rpg significantly outperforms instruction-tuned llms with more parameters and widely adopted rag approaches. technical contributions of this paper can be summarized as follows: ## related work retrieval-augmented generation. retrieval-augmented generation (rag) @xcite ; guu et al., 2020) enhances llms by retrieving relevant passages, thereby improving both the quality and accuracy of generated content, particularly in knowledge-intensive tasks @xcite . early works @xcite chose to retrieve once, incorporating a fixed number of retrieved passages with a query into llms to generate a response. recent research indicates that adaptive retrieval, tailored to the demands of llms, can further enhance generation. flare @xcite uses the generated sentence with a low confidence score as the query to retrieve external knowledge adaptively and then regenerates the current sentence, while self-rag @xcite introduces special tokens allowing the model to adaptively retrieve and reflect the quality of generated content. sure @xcite generates conditional summarizations of retrieval and evaluating them with carefully designed prompts. however, existing approaches may not take full advantage of the planning capabilities of llms. additionally, these methods may struggle to extract relevant content from retrieved passages and are easily influenced by irrelevant information. parameter-efficient fine-tuning. despite the powerful generative capabilities of llms, fine-tuning them requires substantial computational resources @xcite @xcite . to achieve more efficient fine-tuning, parameter-efficient tuning methods have emerged. these methods either fine-tune a small portion of the model parameters or introduce additional learnable parameters without fine-tuning the model itself @xcite @xcite . lora (low-rank adaptation) @xcite reduces the number of parameters to be updated by decomposing the weight matrices into low-rank components. prompt tuning @xcite introduces task-specific prompts by concatenating learnable tokens before the input sequence, requiring minimal parameter updates. multi-task prompt tuning (mpt) @xcite further highlights the commonalities between multi-task learning, suggesting that using a shared soft prompt and task-specific low-rank matrices can yield better results. ## methodology in this section, we first introduce the task definition and basic notation. then, we provide a comprehensive explanation of the rpg framework from the perspectives of fine-grained dataset construction, training, and inference. ## experiments 4.1 experiment setup to validate the effectiveness of our plan-retrieve-generation framework, we conduct in-depth experiments on 5 carefully selected knowledge-intensive tasks. aligning with the previous work @xcite , we conduct zero-shot evaluations and utilize metrics focused on assessing the correctness, factuality, and fluency of outputs. ## conclusion in this paper, we propose a retrieve-plan-generation (rpg) framework, which integrates an explicit plan stage into the lengthy generation process. by generating plan tokens, the model is guided to selectively utilize retrieved paragraphs. the iterative alternation between plan and answer stages ensures that the generated content remains relevant to the topic. to implement this framework, we adopt an efficient multi-task fine-tuning method that equips existing models with both planning and answering capabilities. experimental results demonstrate that rpg outperforms state-of-the-art models across five tasks, validating the effectiveness of our approach.