Tree-of-Prompts: Abstracting Control-Flow for Prompt Optimization (2025)

Abstract: Prompt optimization (PO) generates prompts to guide Large Language Models (LLMs) in performing tasks. Existing methods, such as PromptAgent, rely on a single static prompt, which struggles with disjoint cases in complex tasks. Although MoP uses multiple prompts, it fails to account for variations in task complexity. Inspired by programmatic control flow, we introduce a nested if-else structure to address both varying similarities and complexities across diverse cases. We propose Tree-of-Prompts (ToP), which implements this structure by recursively expanding child prompts from a parent prompt. Sibling prompts tackle disjoint cases while inheriting shared similarities from their parent, and handle cases more complex than the parent. Evaluated on Gorilla (understanding), MATH (reasoning), and a subset of BBH benchmarks, ToP outperforms PromptAgent and MoP, with improvements of 1.4% and 4.6% over PromptAgent and 3.2% and 4.5% over MoP, when tested with GPT-4o-mini and Llama 3.2-3B, respectively.

## introduction large language models (llms) @xcite are used across a wide variety of tasks and the prompt optimization (po) @xcite ) methods need to be able to harness the potential of llms for each of them. we hypothesize that a task consists of diverse cases, varying in terms of 1) complexity, defined as the level of reasoning effort required to solve a case, and 2) disjointness, referring to the degree of separation between cases based on their underlying reasoning strategies. an exemplar is the api function calling task @xcite , where the system is given a question and a list of available functions and objective 2. disjointness : 𝑝 0 𝑡 < 𝑇 : 𝑝 𝑇 : 𝒟 train if-else 𝑝 2 prompt generation prompt generation 𝑝 𝑡 reflection feedback prompt generation : 𝑝 0 & 𝒟 train prompt generation 𝑝 2 & 𝒟 2 𝒟 2 ⊂ 𝒟 train prompt generation 𝒟 1 ⊂ 𝒟 train if-else iterative conditional conditional iterative sequential notation our novelty lies in systematically incorporating both the complexity and the disjointness for po. to this end, inspired by recent work of "programming" via llms @xcite , we formulate po via a control flow graph (cfg) @xcite to offer a unified framework, illustrated in figure 1 (c ). specifically, we integrate the three execution types in cfg: sequential, iterative, and conditional. from the perspective of cfg, promptagent implements the sequential execution via reflection feedback, in conjunction with iterative execution, though it often fails to handle disjoint cases. on the other hand, mop implements conditinoal optimization to address disjoint cases. however, without iterative refinement, each prompt may underfit and remain sub-optimal. integrating all three execution types, we iteratively generate better prompts conditioned on specific cases so that the set of optimized prompts can comprehensively handle diverse cases. ## related work we explore po for tasks with diverse cases, focusing on two objectives: (1) during training, optimiz-ing a comprehensive set of prompts to address both complex and disjoint cases, and (2) during testing, employing case-specific prompts for given inputs, as detailed below. ## tree-of-prompts we introduce a complete cfg to leverage all three execution types, namely, sequential, iterative and conditional. we iteratively optimize multiple prompts with reflection feedback on previous prompts while identifying disjoint cases at each iteration. specifically, as an implementation of cfg for po, we propose tree-of-prompts (top), where the tree structure represents a roll-out of iterative conditional optimization applied to prompts, as illustrated in figure 2(a) . the algorithm for building the tree is presented in algorithm 1. we start with a root node containing the entire training dataset, d train , and the initial prompt, p 0 . the training dataset is defined as d @xmath0, y i )} n @xmath1, where x i and y i denote the input and its corresponding label, respectively, and n represents the number of training examples. for p 0 , we use a basic task description that can be easily crafted by a human, e.g., "given a question and a list of functions, generate function call(s) to answer the question" for api function calling. the algorithm then invokes build(d train , p 0 ), to progressively expand the tree via three execution operations in cfg as follows. conditional execution we first partition d train into k clusters, denoted by {d k } k @xmath2, each corresponding to a specific case, creating k new nodes as the root's children. the prompt for each child node is then conditionally optimized based on d k . note that we do not assume case labels to be known during training, and learn the distribution through our clustering algorithm as detailed in section 3.1. ## conclusion we study prompt optimization for llms to cater to a broad distribution of tasks, which may involve disjoint cases or cases with varying complexity. we propose tree-of-prompts which implements control flow graphs to address both scenarios. in the absence of annotated labels for possible cases, we propose the solution-based clustering method for training examples and case-augmented search prompting for test inputs to identify the target cases at both training and test time. we empirically demonstrate the effectiveness top on gorilla, math, and bbh benchmarks, outperforming strong baselines such as promptagent and mop.