PRACT : Optimizing Principled Reasoning and Acting of LLM Agent (2024)

Abstract: We introduce the Principled Reasoning and Acting (PRAct) framework, a novel method for learning and enforcing action principles from trajectory data. Central to our approach is the use of text gradients from a reflection and optimization engine to derive these action principles. To adapt action principles to specific task requirements, we propose a new optimization framework, Reflective Principle Optimization (RPO). After execution, RPO employs a reflector to critique current action principles and an optimizer to update them accordingly.We investigate the RPO framework under two scenarios: Reward-RPO, which uses environmental rewards for reflection, and Self-RPO, which conducts self-reflection without external rewards. Additionally, we developed two RPO methods, RPO-Traj and RPO-Batch, to adapt to different settings.Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, can effectively learn and apply action principles to enhance performance.

## introduction large language model (llm) agents enable the action execution @xcite @xcite and consecutive reasoning ability @xcite @xcite of llm. specifically, an llm agent has both memory @xcite @xcite and action space @xcite @xcite . adding those information into prompt extends the inference of llm to be multi-turn action execution. therefore, an llm agent is able to decide next actions based on its previous execution observations @xcite @xcite . thought: i should first use search to search for black lounge pants with an elastic waistband. action: search["black lounge pants with elastic waistband"] observation: webpage [item 1] [item 2] . thought: item 2 is most relevant though the color is not available action: click(item 2) ... react agent the order is complete agent reasoning & action search: consider size, color, price, and other attributes, before initiating the search process. be flexible in refining search criteria if initial results do not meet the specified requirements. consider alternative products when the specific product is not available. click: ensure the product meets price, size, color, and other attributes, before proceeding with the purchase. consider adjusting search criteria or exploring alternative products when the specific product is not available. seek technical assistance when encountering recurring errors in action execution. get me black lounge pants with an elastic waistband, and price lower than 30.00 dollars principles pract agent thought: i should first use search to search for black lounge pants with an elastic waistband. action: search["black lounge pants with elastic waistband"] observation: webpage [item 1] [item 2] thought: though item 2 is most relevant, the color is not available. based on my principles, i need to refine my search query. action: search(black lounge pants) optimizing the reasoning framework @xcite @xcite of agent is crucial in generating correct action execution. as of now, customizing an llm agent with existing open-source packages @xcite @xcite requires the designing of action spaces, such as function calls @xcite and code execution @xcite . along with a well-designed agent reasoning framework, i.e. the prompts of agent, an llm is able to consecutively generate correct actions. react @xcite framework achieves wide successes via adding one-step think actions to enhance the reasoning ability of an agent. additionally, reflection @xcite @xcite mechanism is proposed to improve the agent self-correction capability. plan @xcite before execution is also verified to be beneficial. despite many successes, agent execution can fail to make decisions when faced with contradictory observations, particularly during the execution of long-step tasks. to address it, we propose a new type of reasoning strategy, pract, for the llm agent. intuitively, we associate each action with principles that describe the conditions for using that action. during execution, an agent can check these principles before generating the next action. compared to simple action descriptions, principles provide more detailed conditions on when to use the action and offer specific instructions on how to generate the parameters for an action. we demonstrate the benefits of pract in figure 1 via comparing with react agent in webshop @xcite where an agent uses search and click actions to interact with a shopping website. the react agent searches a query and, despite item 2 not having the available color, still clicks it as it appears most relevant. in contrast, the pract agent refines the search based on both search and click principles. consequently, the pract agent decides to search with an improved query, enhancing its decision-making process. to reduce the labor involved in prompt design and to cover more complex scenarios, we propose a new principle optimization framework, reflective principle optimization (rpo). rpo operates in three stages: execution, reflection, and optimization. during the execution stage, an agent performs tasks using predefined or null principles and memorizes the task trajectories. in the reflection stage, the agent reviews its task executions, evaluating how actions were selected and whether they met the task requirements. finally, in the optimization stage, an optimizer refines principles to enhance agent performance. we investigate two optimization methods: rpo-traj, which individually optimizes principles for each trajectory, and rpo-batch, which concatenates all reflections in a batch for optimization. we summarize our contributions as follows: 1) pract is the first work that considers the action principles for llm agent; 2) we propose two optimization methods to adapt the principles to tasks. ## conclusion we propose a novel agent reasoning framework, pract, which provides principles of actions and thus benefits the action understanding of agent. besides, we introduce two optimization algorithm, rpo-traj and rpo-batch for adapting the action principles with task executions. experimental results on four environments demonstrates the effectiveness of pract framework. and the training curve illustrates the learning efficacy of rpo. in conclusion, pract opens a new discussion on how to regularize the agent actions while rpo shades the light on how to optimize the agent prompts.