Devil’s Advocate: Anticipatory Reflection for LLM Agents (2024)

Abstract: In this work, we introduce a novel approach that equips LLM agents with introspection, enhancing consistency and adaptability in solving complex tasks. Our approach prompts LLM agents to decompose a given task into manageable subtasks (i.e., to make a plan), and to continuously introspect upon the suitability and results of their actions. We implement a three-fold introspective intervention: 1) anticipatory reflection on potential failures and alternative remedy before action execution, 2) post-action alignment with subtask objectives and backtracking with remedy to ensure utmost effort in plan execution, and 3) comprehensive review upon plan completion for future strategy refinement. By deploying and experimenting with this methodology—a zero-shot approach—within WebArena for practical tasks in web environments, our agent demonstrates superior performance with a success rate of 23.5% over existing zero-shot methods by 3.5%. The experimental results suggest that our introspection-driven approach not only enhances the agent’s ability to navigate unanticipated challenges through a robust mechanism of plan execution, but also improves efficiency by reducing the number of trials and plan revisions by 45% needed to achieve a task.

## introduction two roads diverged in a yellow wood, and sorry i could not travel both • • • then took the other, as just as fair, and having perhaps the better claim robert frost the enduring appeal of frost's emblematic poem, "the road not taken," resides not just in its poetic elegance, but also in the profound lesson it imparts about decision-making. as we stand at the crossroads of a choice, it is a daunting challenge to assess probable outcomes and choose a course that best aligns with our objectives. this task becomes even more formidable when large language model (llm) agents @xcite @xcite have to navigate complex scenarios unfolding in real time, e.g., solving tasks in web environments @xcite @xcite , conducting simulated science experiments @xcite , and solving embodied household tasks @xcite . indeed, llm agent decision-making has witnessed enhancement by post-hoc reflection and correction @xcite , coupled with adaptive planning @xcite , where the agents learn from past successes and failures while concurrently mapping out flexible strategies. however, reflection usually works sequentially where only one hypothetical error can be corrected for each head-to-toe execution trajectory. considering that such reflection is a test-time strategy, it poses a great efficiency issue. for instance, the agent could retry 10 times before concluding it still can not solve the task. furthermore, self-reflection involves frequent shifts in plans which, albeit a mere inconvenience for humans, can lead to disorientation for ai agents. this may produce confusion, a standstill, or even an infinite loop of failure, which substantiates the importance of thoroughly executing a set plan with utmost effort before resorting to a plan revision. therefore, this paper puts forward a methodology aimed at achieving an optimal balance between consistency and adaptability. this critical equilibrium mirrors the resilience and agility that is anticipated of a capable system that is prepared for curveballs but unwavering in the execution of its plan. fig. 1 highlight our design in comparison to existing reflection strategy. in this paper, we introduce a novel approach that integrates introspection into the fabric of llm agents. this approach enables agents to continuously reflect on their actions, thereby stimulating a learning process that dynamically optimizes exploration paths and enhances robust decision-making under uncertainty. our introspective intervention focuses on three principal dimensions: in this paper, we develop and expand upon several key themes within the realm of natural language processing, with a specific focus on the integration of action generation, planning, and reflection in the construction of llm agents. action generation llms have been employed in tasks requiring decision-making or action generation and have proven useful as agent-controlling policies in embodied environments @xcite @xcite . they have also demonstrated effectiveness in text-based environments @xcite @xcite , where techniques like react @xcite have shown notable benefits. despite its success, re-act's limitation lies in its inability to adjust to changes in the environment. several improvements @xcite have been proposed to counter these limitations, advocating for self-reflection to enhance decision-making and reasoning. however, these techniques primarily aim to improve single plans or trajectories without considering alternative actions, which could modify the plan in a wrong direction. position bias mitigation while comparing answer choices is generally effective, large language models used for action generation are not without flaws. they can exhibit bias, especially towards the first (or sometimes second) answer they see, regardless of its quality. this is known as position bias @xcite . our method mitigates this bias by asking follow-up questions that challenge its own answer. planning extensive research has explored the potential of llms in task planning @xcite @xcite @xcite . the concept of decoupling planning and execution in formulating llm agents has been validated through numerous paradigms such as rewoo @xcite , adapt @xcite , structured self-reflection @xcite , and defs @xcite . nonetheless, these methods exhibit a deficiency in establishing a resilient mechanism for plan execution, with agents frequently revisiting and revising their plans following each instance of adverse environmental feedback, often due to inaccurately executed actions. our approach, conversely, emphasizes executing a previously defined plan with unwavering effort before considering any modifications. this guarantees a more stable and consistent problem-solving process. to implement this, the factor of tree search becomes crucial for exploring the best solutions. past approaches, including tot @xcite , rap @xcite , lats @xcite , adaplanner @xcite , and toolchain* @xcite , have incorporated tree search techniques in identifying the optimal route to the desired solution. however, our approach distinguishes itself by engaging the llm in preparing alternate solutions in anticipation of impending failures, ensuring more comprehensive consideration in action generation. reflection and self-refinement reflection and refinement techniques have advanced significantly through works such as reflexion @xcite , adaplanner @xcite , and auto-eval @xcite . our methodology further enhances this by incorporating an anticipatory reflection mechanism that operates before each action rather than performing post-hoc reflection after each complete trial. this approach simplifies exploration by expediting remedial action and reducing extensive backtracking and serial plan revisions, thereby improving the overall efficiency. ## method given a task t and an environment e with which the llm agent g interacts, our objective is to enable the agent to systematically and adaptively complete the task through introspective methods. we first present how we decompose the task and generate action regarding each state in the environment in §3.1 and §3.2. then we introduce the introspection mechanism in §3.3. ## experiments in this section, we demonstrate how introspection enhances consistency and adaptability of llm agents in solving complex tasks in web environments. we first introduce the experimental setup for evaluation ( §4.1), followed by evaluation results ( §4.2). detailed error analysis is provided in §5, which highlights the directions for future endeavor. ## error analyses the subsequent sections shed light on an analysis of errors we observed from the agent's behavior when executing tasks. two key areas have been identified for detailed discussion: an agent's occasional inability to fully learn from past failures, and inefficiencies in solving specific kinds of tasks due to a sequential planning scheme. ## conclusions in this work, we introduce a novel introspective methodology that significantly enhances the problem-solving capabilities of llms in complex environments, as demonstrated through comprehensive evaluations in the webarena setting. our approach strategically decomposes tasks into actionable subtasks and incorporates a three-tiered introspection process, which includes anticipatory reflection, robust post-action evaluation, and episodelevel plan revision. this setup not only allows llm agents to adapt their strategies in real time but also fosters long-term learning, reducing the need for frequent interventions as experience accumulates. the application of our introspective agent design in the webarena benchmark demonstrates substantial performance gain (3.5%) over state-of-the-art zero-shot approach, along with sta-ble performance curve with increasing number of rounds. such benefits are accompanied by almost halving the number of plan revisions (45%) during error handling. in summary, by enabling llm agents to proactively contemplate potential failures, evaluate actions post-execution, and continuously refine their strategy based on experiential insights, our approach equips ai systems with a human-like strategic thinking capability.