P lan RAG : A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers (2024)

Abstract: In this paper, we conduct a study to utilize LLMs as a solution for decision making that requires complex data analysis. We define **Decision QA** as the task of answering the best decision, d_{best} , for a decision-making question Q , business rules R and a database D . Since there is no benchmark that can examine Decision QA, we propose Decision QA benchmark, **DQA**. It has two scenarios, Locating and Building, constructed from two video games (Europa Universalis IV and Victoria 3) that have almost the same goal as Decision QA. To address Decision QA effectively, we also propose a new RAG technique called the *iterative plan-then-retrieval augmented generation* (**PlanRAG**). Our PlanRAG-based LM generates the plan for decision making as the first step, and the retriever generates the queries for data analysis as the second step. The proposed method outperforms the state-of-the-art iterative RAG method by 15.8% in the Locating scenario and by 7.4% in the Building scenario, respectively. We release our code and benchmark at https://github.com/myeon9h/PlanRAG.

## introduction in many business situations, decision making plays a crucial role for the success of organizations @xcite . here, decision making involves analyzing data, ultimately leading to the selection of the most suitable alternative to achieve a specific goal @xcite . for example, we assume that one of the goals of the pharmacy company "pfizer" is to minimize the production cost while maintaining on-time delivery from plants to customers in the pharmaceutical distribution network @xcite , and the production cost is proportional to the amount of operation time and in general, the decision-making task requires performing the following three steps: (1) making a plan for which kind of analysis is needed for decision; (2) retrieving necessary data using queries; to achieve the goal, we propose, decision qa, a new decision-making task for language models. decision qa is defined as a qa-style task that takes a pair of database d, business rules r and a decision-making question q as input and generates the best decision as output. figure 1 shows a situation in europa universalis iv game where countries compete in trade at the age of discovery, as an example of decision qa. each country decides which trading city (i.e., node) it should locate a merchant on to maximize its profit on its main (i.e., home) trading node. the example shows that a decision-making llm decides to locate a merchant in doab to maximize the profit of deccan, the home trading node of the country bah, after analyzing the database about international trade. next, we propose a benchmark for decision qa called dqa, which consists of the following two scenarios: locating and building. the former scenario consists of decision questions like "which tradingflow table tradingnode table nodecountry table country table figure 1: example of decision qa. a red dot represents a trading node. a profit in the deccan box indicates a potential profit change by each decision. note that the potential profit changes are not in the database, which should be calculated from the database. each country has only a single main(home) trading node. the underlined column names in a table indicate the key of the table. the recent breakthrough in llms is making it possible to replace steps (2) and (3) of the decision making task with llms, in particular, based on retrieval-augmented generation (rag) technique. so far, a lot of rag-based methods have been proposed for various tasks @xcite @xcite @xcite @xcite . in these methods, a retriever finds external data highly relevant to a question and conveys it to lms, so that lms can generate an answer based on it @xcite . recently, the iterative rag technique has also been proposed to address more complex problems that should utilize retrieved results to perform further retrievals @xcite . however, the existing rag-based methods mainly focus on the knowledge-based qa tasks @xcite , but do not focus on the decision making qa task. as a result, they are not very good at handling step (1), i.e., making a plan for decision, in our observation. for instance, in figure 1 , an lm for decision making should reason which analysis is needed to perform for maximize the profit of deccan. however, the existing methods just try to identify, for example, what deccan is. to address this limitation, we propose the iterative plan-then-retrieval augmented generation technique, planrag, which extends the iterative rag technique for decision qa. a planrag-based lm first makes a plan for which kind of analysis it needs by examining data schema and questions (the planning step). next, it retrieves the scattered pieces of data for the analysis by generating and posing queries (the retrieving step). finally, it assesses whether it needs to make a new plan for further analysis, and then repeats both the planning and retrieval steps iteratively (the re-planning step), or makes a decision based on the data (the answering step). to validate the effectiveness of our planrag on decision qa, we compare both the state-of-the-art iterative rag-based lm and our planrag-based lm for the dqa benchmark. our contributions are summarized as follows: ## decision qa task we define decision qa as the task of answering the best decision d best given a decision-making question q, business rules r, and structured database d that follows a schema s. here, q contains a textual goal that users want to be achieved through the decision d best , and r contains textual descriptions of formulas that are referenced to reason d best . without loss of generality, the database d is too large to fit in the input of an lm all at once. therefore, we assume that an lm retrieves data from d by posing a query for data analysis, which we call as a data analysis query hereafter. we assume that d is either a labeled property graph (lpg) database or a relational database (rdb). for example, the database in figure 1 is a relational database. here, an lpg refers to a graph with properties on edges and nodes, which is widely used in the industry @xcite . we call a labeled property graph database as simply a graph database, or gdb hereafter. ## methodology: planrag for decision qa, the existing rag technique @xcite ) tries to answer the best decision d best for given ⟨q,s,r⟩ through a single type of reasoning that utilizes on the results retrieved from d by data analysis queries. figure 3 (a) shows its reasoning process. if the retrieval from d is performed only once, the process is called as single-turn rag. otherwise, if the retrieval is performed multiple times, the process is called as iterative rag. in contrast, our iterative plan-then-retrieval augmented generation (planrag) technique tries to answer d best through two types of reasoning. the first type of reasoning is making a plan, and the second type of reasoning is similar to the reasoning of the existing rag, i.e., answering based on the results retrieved from d by of data analysis queries. in particular, we built a single lm that can perform both types of reasoning because to reduce the side effects of using separate lms. we prompted the lm by adding the 'plan' and 're-plan' instructions to @xcite , about which details are in appendix. a.6.2. figure 3 (b) shows the reasoning process of planrag, where we explain the steps of (1) planning, (2) retrieving & answering, and (3) re-planning, in detail as follows. planning: in this step, an lm takes ⟨q, s, r⟩ as input, and then generates an initial plan for data analysis. the initial plan describes a series of data analyses that needs for decision making and so need to be performed in the retrieving step. the red box in figure 4 (b) shows its example. retrieving & answering: unlike the previous rag technique, an lm takes not only ⟨q, s, r⟩ but also the initial plan as input. then, it can generate data analysis queries for decision making much more effectively than the previous rag. figure 4 shows how planrag-based lm generates the queries differently from the previous rag. the queries are actually executed by sql or cypher to the database through rag interfaces such as langchain foot_0 and llamaindex foot_1 . the query results are used iteratively for reasoning about whether it needs re-planning or just a further retrieval for better decision making. through the backward link to the planning process, the planning and retrieving processes are iteratively performed until an lm determines that there is no longer a need for further analysis to make a decision. re-planning: re-planning is done when the initial plan is not good enough to solve the decision question. in order to make the lm possible to decide whether re-planning or not, we prompted the lm with some instructions to assess the current plan by referring to the result of each retrieval step (see appendix. a.6.2 for details). as a result, the lm takes not only ⟨q, s, r⟩ but also a current plan and query results as input and generates a new plan to do further analysis, or correct the direction of previous analysis. question: where should i locate my merchant? my goal is maximizing bah's profit on home node, deccan. tradingnode tradingflow development home node country country nodecountry schema doab, patagonia, malacca db retrieving i need to find all the source nodes of "deccan". deccan: @xmath0.906, @xmath1.821, … doab: @xmath2.978, @xmath3.864, … db to calculate the profit, i need to know the local_value, ingoing, country's trading power, and total trading power of each node. i also need to know the development of the country "bah" to calculate the added value if the node is inland. doab: @xmath4.978, @xmath5.864, … patagonia: @xmath6.082, @xmath7.0, … malacca: @xmath8.652, @xmath9.107, db the observation seems to be incorrect. i was expecting to get the details of the source nodes of "deccan", but i got the details of "deccan" itself and "doab". i need to correct my query to get the details of the source nodes. you should allocate the merchant to the "patagonia" planning step 1: identify the trading nodes that are source of "deccan". step 2: calculate the potential profit increase for each of these nodes if a merchant is placed there. step 3: identify the node with the highest potential profit increase doab db retrieving (step 1) i need to identify the trading nodes that are source of "deccan". 8.288 db (step 2) the observation shows that "doab" is the only trading node source of "deccan". now, i need to calculate the potential profit increase for "doab" if a merchant is placed there. now that i have the details of the source nodes, i can calculate the potential profit increase for each node if a merchant is placed there. doab: 0.0011, patagonia: 0.0728, malacca: 0.0017 db answering you should allocate the merchant to the "doab" doab db (step 3) the potential profit increase for "doab" if a merchant is placed there is approximately 8.29. now, i need to identify the node with the highest potential profit increase. (a) iterative rag (b) planrag answering has merchant 𝑻𝑷 𝒄𝒐𝒖𝒏𝒕𝒓𝒚 is home country trade node 𝑻𝑷 𝒕𝒐𝒕𝒂𝒍 ov iv lv is inland trade node flow destination source cated on the following two reasons. first, in most of real-world business situations, it is very hard to know the strategies for making the best decisions in advance. second, in few-shot settings, we have observed that an lm is overfitted to not only the problem-solving strategy but also the content of a database in the given shots. an answer from a decision maker is considered correct only if it is semantically identical to a ground-truth best decision on dqa. for example, in figure 4 (b), we consider the answer is correct since the ground-truth best decision is doab. ## conclusions in this paper, we explored the capability of llms as a solution for decision making. we proposed the new decision-making task, decision qa, which answers the best decision for a given complex decision-making question that requires considering both the business rules and business situation represented in a large database (in either rdb or gdb). we built the benchmark for decision qa, called dqa, by extracting 301 sets of a database (in both rdb and gdb), a question, and an answers(ground truth) from two popular video games imitating real business situations that require decision making. we also proposed the new rag technique called planrag, which performs planning before retrieving and re-planning if the initial plan is not good enough. through extensive experiments, we demonstrated that planrag significantly outperforms the sota iterative rag for the decision qa task.