PEARL : Prompting Large Language Models to Plan and Execute Actions Over Long Documents (2024)

Abstract: Strategies such as chain-of-thought prompting improve the performance of large language models (LLMs) on complex reasoning tasks by decomposing input examples into intermediate steps. However, it remains unclear how to apply such methods to reason over long input documents, in which both the decomposition and the output of each intermediate step are non-trivial to obtain. In this work, we propose PEARL, a prompting framework to improve reasoning over long documents, which consists of three stages: action mining, plan formulation, and plan execution. More specifically, given a question about a long document, PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE, FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain the answer. Each stage of PEARL is implemented via zero-shot or few-shot prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate PEARL on a challenging subset of the QuALITY dataset, which contains questions that require complex reasoning over long narrative texts. PEARL outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of PEARL is critical to its performance. Overall, PEARL is a first step towards leveraging LLMs to reason over long documents.

## introduction performing complex reasoning over long input documents often requires forming high-level abstractions of the text (e.g., plots and themes in a narrative) and then conducting a variety of inferences on top of those abstractions @xcite . consider the following question about the story "breakaway" from the quality dataset @xcite : each stage in pearl is achieved via zero-shot or fewshot prompting of an llm (in our work, gpt-4). we also provide example outputs from each stage. what part of the final scene best connects to the story's opening conversation? to answer this question, we need to gather and synthesize information from across the story, which motivates decomposing the question into a plan of actions, as in: given the rapidly advancing capabilities of large language models (llms), how can we use them to answer questions like these? while we could directly prompt llms to generate the answer, prior work on simpler reasoning-based tasks shows that this method is inferior to chain-of-thought prompting @xcite , which encourages the llm to provide step-by-step explanations and intermediate outputs before producing the answer. unfortunately, cot is not well-suited for tasks involving complex reasoning over long input documents, as both the decomposition of the original question and the intermediate outputs of each step are non-trivial to obtain, as in the above example. given the difficulty of obtaining plans and intermediate explanations for long documents, one potential solution is to delegate this task to smaller executable modules instead of forcing the llm to come up with all of them at once. in this work, we introduce pearl, a framework that combines planning with executable actions for reasoning over long documents. each stage of pearlaction mining, plan decomposition, and plan execution -is implemented by applying zero-shot or few-shot prompting to an llm. the stages (figure 1 ) can concisely be described as follows: ## related work our work builds on recent llm prompting research and also connects to work on reasoning over long documents. before describing pearl, we first survey related papers to contextualize our work within this fast-moving field. prompting methods: recently, the capabilities of large language models @xcite @xcite have significantly increased as a result of learning from instructions or feedback @xcite @xcite to better align their outputs to human preferences. when provided with well-crafted prompts, such as chainof-thought @xcite explanations, these state-of-the-art models exhibit impressive reasoning abilities. a plethora of new prompting techniques ( reasoning over long documents: large language models have showcased remarkable reasoning capabilities @xcite , including mathematical reasoning @xcite , commonsense reasoning @xcite , and symbolic reasoning @xcite . most of these tasks do not involve long context inputs, and thus they are able to benefit from few-shot in-context cot prompting. in this paper, we primarily focus on tasks that contain long input contexts @xcite @xcite , specifically generative question answering based on long input articles. to address the absence of reliable evaluation for long-form qa @xcite , @xcite proposes automatic metrics for evaluating the correctness of the answer, whereas in this work, we use llm-based evaluation by taking advantage of the multiple-choice setup of existing qa dataset. prior to the shift to prompting-based methods, approaches including contrastive learning-based sequence-level objectives @xcite , iterative hierarchical attention @xcite , and joint modeling of machine reading and answer generation @xcite have been employed to enhance long-context question answering. ## pearl: planning and executing actions for reasoning over long documents we are interested in using llms to solve tasks that require complex reasoning over long documents. 2 in this paper, we focus on the task of answering questions about long-form narratives. most prompting strategies that aim to improve the reasoning 2 as there is no consensus on what is "long", we consider it to mean documents of several thousands of tokens in length. ## main results we discover that pearl significantly outperforms competing prompting methods on questions that require reasoning over long contexts, which demonstrates the utility of the planning module. we also observe a small drop in accuracy on questions that require only short contexts, possibly because the plans end up over-complicating what is a simple reasoning process. in this section, we dig deeper into the main results of our experiments, which are presented in pearl improves accuracy on long-document qa: overall, pearl's accuracy is higher than that of all competing methods, particularly for the quality split annotated by humans as requiring long contexts to answer @xcite . furthermore, we observe in figure 6 that for questions marked by quality workers as requiring the longest possible context, pearl improves substantially compared to the zero-shot gpt-4 baseline (72.4% vs 61.9%). our method's slightly diminished performance on the short split is likely due to both "overthinking" these simpler questions, as well as error propagation from plan execution steps as highlighted in section 6. finally, we point out that all methods achieve higher accuracies on the short split compared to the long split, indicating the challenging nature of this set of questions. number of actions impacts performance: in figure 7 , we show that the size of the action set is an important factor in pearl's performance. with just a single action (i.e., execute a free-form natural language instruction), foot_4 pearl's accuracy on the long subset drops to 64%. with too many actions (140 in the plot), its accuracy also degrades, likely because the action space is too fine-grained for the model to properly execute all actions. we note that the optimal number of actions likely differs from task to task, so it is an important hyperparameter to consider when tuning pearl. action execution is necessary: do we actually need to execute the generated plans to answer these questions? feeding just the generated plan to the model along with the question (minus any execution results) may still encourage the llm to follow the plan's reasoning steps and generate a better answer. however, we observe that removing the execution results from the model's input reduces absolute accuracy by around 3 points, which suggests that it is important to perform multiple passes over the document to execute each action before answering the original question. with that said, we do observe a modest improvement over the gpt-4 zero-shot and cot baselines (∼ 2 absolute points), which suggests that the plan itself is also valuable. ## analysis in this section, we analyze the behavior of pearl by diving into the composition of its generated plans, its most preferred actions, and what types of questions it improves most on. we also offer a qualitative error analysis as well as a human evaluation on the correctness of the generated plans. plan statistics: plans are roughly 4 actions long on average, with around 3.4 unique actions per plan. the most commonly used actions are shown in figure 8 . apart from the string concatenation action concat, the most frequently used action is find_character, which can be convenient for understanding long literary text. other less often used actions cover both those that can transfer across domains, e.g., compare, and those specific to narrative understanding, e.g., find_emotion. accuracy by reasoning types: since quality questions require different reasoning strategies to solve, what types of reasoning does pearl help improve the most? to this end, we further evaluate questions based on the type of reasoning required to answer them. 10 pearl is significantly slower than zeroshot prompting: the improved performance of pearl comes at the cost of longer running time and cost: pearl requires 4.4 times more tokens in the prompt, and it needs to generate 1.3 times more tokens owing to the intermediate steps. 11 specific examples where pearl helps: to better understand pearl, we qualitatively analyze 40 examples for which zero-shot gpt-4 generates incorrect answers while pearl answers correctly. this analysis reveals two key advantages of pearl. first, while zero-shot prompting is reasonably good at finding salient information from the 10 we prompt gpt-4 with the definition of each reasoning type presented in the appendix @xcite and ask it to label each question with up to two reasoning types. 11 these multiples were estimated from a small run of 30 examples. where does pearl go wrong? we additionally examine 40 examples for which pearl answers incorrectly, and group the errors into three categories (detailed examples in appendix a ## conclusion in this work, we introduce pearl, a framework for tackling complex reasoning over long documents. to answer a question, pearl first proposes a plan based on a set of actions mined from a training set, and then it executes the plan step by step via prompting itself with a template filled with output from previous stages. we demonstrate the effectiveness of pearl on a challenging subset of qual-ity. experiments and analysis show that prompting gpt-4 with pearl yields more accurate and comprehensive answers than zero-shot and chainof-thought prompting, and human annotators judge the generated plans to be reasonable.