Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization (2024)

Abstract: Large Language Models (LLMs) exhibit robust problem-solving capabilities for diverse tasks. However, most LLM-based agents are designed as specific task solvers with sophisticated prompt engineering, rather than agents capable of learning and evolving through interactions. These task solvers necessitate manually crafted prompts to inform task rules and regulate LLM behaviors, inherently incapacitating to address complex dynamic scenarios e.g., large interactive games. In light of this, we propose Agent-Pro: an LLM-based Agent with Policy-level Reflection and Optimization that can learn a wealth of expertise from interactive experiences and progressively elevate its behavioral policy. Specifically, it involves a dynamic belief generation and reflection process for policy evolution. Rather than action-level reflection, Agent-Pro iteratively reflects on past trajectories and beliefs, “fine-tuning” its irrational beliefs for a better policy. Moreover, a depth-first search is employed for policy optimization, ensuring continual enhancement in policy payoffs. Agent-Pro is evaluated across two games: Blackjack and Texas Hold’em, outperforming vanilla LLM and specialized models. Our results show Agent-Pro can learn and evolve in complex and dynamic scenes, which also benefits numerous LLM-based applications.

## introduction designing a human-level agent with robust problem-solving abilities has long been a vision in the academic community. this necessitates the agent to possess learning and generalization capabilities across a diverse array of tasks. the advent of large language models (llms) @xcite figure 1 : for interactive tasks, e.g., imperfectinformation games, we propose a versatile agent framework capable of self-learning and evolving. firstly, our agent constructs beliefs about itself and the environment. then it autonomously updates its prompts through policy-level reflection on past trajectories and beliefs, evolving a better behavioral strategy. @xcite @xcite @xcite @xcite has shed light on this vision, especially they can be rapidly generalized across a wide range of tasks with only a few demonstrations @xcite . benefiting from this, many systems built upon llms have showcased markedly enhanced performance such as questionanswering @xcite @xcite @xcite , code generation @xcite , and real-world application @xcite . despite these achievements, building a humanlevel agent remains a challenging endeavor. first, most llm-based agents are designed for specific tasks through sophisticated prompts, including detailed task descriptions and behavioral specifica-tions. however, numerous real-world tasks, e.g., business, company negotiations, and security, are more intricate with imperfect information, necessitating laborious efforts to design strategic behavior. second, most llm-based agents do not consider interacting with task scenarios, and more critically, cannot learn from past experiences and evolve their behavioral strategies during interactions. in contrast, humans often learn and adjust their behaviors through interaction, especially in novel scenarios. in light of these, a promising yet under-explored topic emerges: can llm-based agents learn and elevate behavioral strategies by interacting with the environment like humans? it should be an indispensable ability of a human-level agent. recently, numerous studies @xcite @xcite @xcite undertake intriguing explorations, e.g., utilizing feedback for self-correction at the action-level. besides, several efforts also explore deploying llm in interactive games, including starcraft @xcite , minecraft @xcite , strategy-based gaming @xcite @xcite . similarly, we first evaluate llm-based agents with the self-correction strategy in dynamic interactive scenarios, such as multi-player texas hold'em, which is a zero-sum game with imperfect information. however, we observe that it loses most of the rounds to its opponents, even the most advanced llms. upon examining its reasoning thoughts and actions, we find that it often adopts irrational behaviors and is unable to deduce effective strategies from long action sequences. to answer the above question, the theory of mind (tom) @xcite ) may provide some insight. in this framework, each human develops perceptions of himself (self-belief) and the external environment (social-belief) in the social context, and then grounds their decisions on these beliefs, or adjusts incorrect beliefs in response to external feedback. inspired by this, we advocate agent-pro: a llm-based agent with policy-level reflection and optimization. agent-pro is endowed with the capacity to learn and evolve within environments, i.e., autonomously reflect on past experiences, calibrate its beliefs about itself and the environment, and optimize its behavior policy without parameter tuning. concretely, as shown in figure 1 , an llm-based agent involves an llm as the foundational model and some instructions in the prompt to regulate its behavior (policy). upon observing partial information from the scenarios, agent-pro first updates its self-belief and world-belief, then makes decisions based on these beliefs. after exploring tasks, agent-pro performs a policy-level reflection and optimization on past trajectories, beliefs, and results. it autonomously "fine-tunes" its beliefs, searches for useful prompt instructions, and consolidates them into a new behavior policy. the experiments in two zero-sum games, blackjack and texas hold'em, demonstrate that agent-pro, after evolution, can defeat vanilla llms and specialized models, improving the game's payoffs. it indicates that agent-pro enhances its capabilities through interaction and reflection without human guidance. as depicted in figure 1 , the initial prompt is quite simple (left bottom), but after learning and evolution, the agent-pro generates many practical instructions (right bottom). for instance, agent-pro records estimations of each opponent's style in task description and adds specific goals, strategies in behavior policy. our agent-pro is different from previous strategies, like @xcite . firstly, policy-level reflection is designed for policy updating in long-horizon tasks. it is aimed at longhorizon policy updating rather than immediate action correction. the input is a sequence of actions and delayed feedback, while the output is an optimized strategy, rather than a specific action. therefore, policy-level reflection corrects irrational beliefs and optimizes the old policy into the new one. as introduced in section 3.2, our policy-level reflection includes belief calibration, policy updates by refining behavioral guidelines and world modeling, and policy verification. besides, we innovatively distill long-term memory into behavioral guidelines and world models through prompt optimization. most previous strategies store historical experience as verbal long-term memory and use it for text-based reasoning. in contrast, we further construct an optimizable policy from long-term interactions, i.e., behavioral guidelines and environmental models. this includes self-summarized game objectives and rules, effective strategies derived from reflection, and demonstrative trajectories. the contributions of our work are as follows: not just in card games, similar scenarios abound in the real world as well. through self-learning and evolution, agent-pro can enhance deployment effectiveness in those scenarios, expanding the capability boundaries of llm-based agents notably. ## problem definition our study focuses on multi-player imperfect information interactive games, with two characteristics: imperfect information. unlike perfect information games (e.g., chess), imperfect information scenarios are characterized by agents only having access to their own states and public information, without knowing the states of others, e.g., in texas hold'em, players cannot observe others' cards, which is dissimilar to many llm-based tasks. ## methods to empower agents to learn in interactive contexts, a typical method is reinforcement learning @xcite . this involves exploring highly rewarding actions through trial and error and solidifying these experiences into model parameters. nonetheless, the training overhead for llms is substantial. therefore, we employ a gradient-free "exploration-learning" strategy that enables llm-based agents to learn through in-context learning. specifically, we convert the policy learning into a prompt optimization process, i.e., llm autonomously reflects and updates the prompt's instructions based on its exploration experience, solidifying the high-reward strategies into the prompts. benefiting from llm's generalization capabilities, our agent can summarize rules and learn specialized skills from a small number of samples like humans, making it well-suited for many real-world scenarios. as shown in figure 2 , agent-pro comprises three components: (1) a belief-aware decision-making process. it first updates beliefs about the world and itself, rendering more coherent and consistent decisions in dynamic and imperfect game scenarios. (2) a policy-level reflection. rather than reflecting on a single action, our design empowers llms to self-reflect on irrational beliefs from failed experiences. then, it summarizes these erroneous beliefs into specific prompt instructions, like acting strategy (behavioral guideline), descriptions of the task world, and conjectures about other players (world modeling), etc, which can calibrate its incorrect beliefs, evolving into a better policy. ## discussions beyond card games, agent-pro holds the potential to handle other complex tasks. although we have only tested agent-pro in blackjack and texas hold'em tasks, we believe there is great potential for it to be generalized to many complex tasks. ## conclusion we design an llm-based agent, agent-pro, capable of learning and evolution in complex interactive tasks. it first constructs a dynamic belief for decision-making in uncertain scenarios. then agent-pro reflects on its interactive experiences, corrects irrational beliefs, and summarizes its reflections into two instructions: behavioral guidelines and world descriptions for a new policy. lastly, we evaluate agent-pro in two zero-sum games and observe that its decision-making capabilities significantly improve after learning from historical experiences.