A Zero-Shot Language Agent for Computer Control with Structured Reflection (2023)

Abstract: Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal in a live computer environment (e.g. MiniWoB++). To perform a task, recent works often require a model to learn from trace examples of the task via either supervised learning or few/many-shot prompting. Without these trace examples, it remains a challenge how an agent can autonomously learn and improve its control on a computer, which limits the ability of an agent to perform a new task. We approach this problem with a zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iteratively progresses a task by identifying and learning from its mistakes via self-reflection and structured thought management. On the easy tasks of MiniWoB++, we show that our zero-shot agent often outperforms recent SoTAs, with more efficient reasoning. For tasks with more complexity, our reflective agent performs on par with prior best models, even though previous works had the advantages of accessing expert traces or additional screen information.

## introduction prior works have shown promises in using large language models (llms) for action generation, e.g. saycan @xcite , react @xcite , toolformer @xcite , and swiftsage @xcite ) over a variety of live environments (e.g. miniwob++ @xcite , alfworld @xcite , and alphacode @xcite . a shared approach is to use llms to follow expert traces, comprehend environment changes, plan future actions, and execute an action by composing api calls; all in the form of text. some works have shown that iteratively attempting a task with : work done during internship at google research. recently, miniwob++ has been used as a testbed for llm's capacity at modularized computer tasks. to learn a task, a common approach is the use of extensive trace examples of the task for direct supervision (e.g., @xcite , webgum @xcite ), self-supervision @xcite , or few/manyshot prompting (e.g., rci @xcite , synapse @xcite ). they have achieved more than 90% task completion rate on dozens of computer tasks, seemingly to have solved the computer control problem. however, the requirement of expert traces for learning to perform a task limits the agent's ability on new tasks. without using carefully selected traces as guidance, can an agent autonomously learn and improve its control on a computer? to address this question, we propose a zero-shot agent. we build our agent on top of palm2 @xcite , a recent llm, and our agent employs a unified instruction prompt set across different tasks, without extensive tailoring for individual tasks. in addition, recent works, e.g., rci @xcite , adaplanner @xcite , and synapse @xcite , utilize a screen representation that may include much more information than what is presented to a user on the screen. for instance, fig. 1 shows an example of elements that are not shown on the screen yet present in the html that is fed to the llm. using such additional information arbitrarily reduces the difficulty for the agent to perform the task. yet in general use cases, such information might not be readily available, and relying on such information can potentially hamper the applicability of the agent. we manually examined 13 relatively challenging tasks on miniwob++ that are supposed to span multiple screens, and found 5 of them contained such information-multi-screen information in a single observation-in their html. our contributions are as follows: firstly, we employ a compact screen representation that assumes much less information than what is used by previous works, thus resulting in a more general and realistic test environment. secondly, we propose a simple yet efficient action planner that can accurately plan out executable actions on a state in one pass. with recent capacity of llm, we show that such a "naive" strategy can solve almost all the easy tasks on the miniwob++ benchmark. for more challenging tasks, we take inspiration from reflexion @xcite and propose a structured thought management strategy to facilitate reflection, allowing the agent to effectively learn and improve from exploration failures. with a few rounds of attempts, our agent achieves comparable performance with prior few/many-shot state-of-the-art. to the best of our knowledge, our agent is the first zero-shot design for computer control tasks foot_0 . ## background llms have become an emerging tool for planning and executing necessary steps to fulfill a top-level goal. these models have exhibit high capacity to follow in-context traces to solve complex tasks. planning & reflection react @xcite used intermediate thoughts to guide longchain of action planning in a live environment. beyond one trial of planning, reflexion @xcite and self-refine @xcite recently found the ability of llm to selfcriticize and self-improve can iteratively learn and solve a given task via multiple trials. nevertheless, recent planning works require extensive and customized trace prompts for the llm agent to learn accurate planning. swiftsage @xcite reduces the extensive planning calls to llm by training a small and fast planning module to facilite long-chain of actions. in a similar motivation, our zero-shot agent is based on an efficient staged planner. we list a detailed comparison in tab. 1. language/vision agent for computer control miniwob++ @xcite has several dozens of fundamental computer tasks hosted as live environments. recent advances on this benchmark have benefited from extensive human annotation to facilitate behavior cloning and reinforcement learning, such as cc-net @xcite and pix2act @xcite . beside these models that rely on multimodal or vision input, another track is using llms as an off-the-shelf agent, and use prompted inference for action generation, such as rci @xcite , adaplanner @xcite , and synapse @xcite . we highlight our technical differences in tab. 2. ## environment interface the role of a language agent is to comprehend screen representations (sec. 3.1&3.2), execute actions on the environment (sec. 3.3), and react to environment feedback (sec. 3.4). ## planning strategies in this section, we summarize the planning strategies used in recent llm-based planning works to motivate our staged planning. with a given goal, an agent model is to issue actions based on prior interaction with the environment. for brevity, let us denote the interaction as a sequence of state and action pairs ps i , a i q. ## structured self-reflection in practice, a high-level human instruction or goal can be ambiguous, and an environment can be partially hidden. therefore, agents are prone to making mistakes; this is even true for a human user when executing a task @xcite . once a negative signal is given, such as no change or failed (sec. 3.4), we ask the agent to reflect on its past action trajectory, suggest an improved version, and then retry the task. an example of successful reflection from our agent is shown in fig. 3 . in recent works @xcite , reflection is conducted at the end of each trial by accumulating a text entry. the entry is essentially a natural language statement about what should have been done instead. at trial t, the amount of entries maintained in the reflection memory is limited by multiple factors. for one, it increases the input length of llm. moreover, it requires the llm agent to handle the thought structures through multiple trials. in practice, @xcite limited the memory size p r1, 3s. ## experiments we start with categorizing tasks by their planning complexities to have an isolated testbed. then we experiment with our staged planning in sec. 6.3-6.4. finally, we examine if our zero-shot agent can learn from mistakes in sec. 6.5. our prompts are in appx. a-e. complete results are in appx. f. ## conclusions we proposed the first zero-shot agent for computer control tasks. our agent design generalizes the workflow for easy and complex tasks via efficient planning and structured thought management. we evaluated our agent on the miniwob++ benchmark, showing that our agent, with often one pass of planning query, outperforms the best iterative planning agent as well as supervised state-of-theart on simple tasks. for complex tasks, we show that our agent design performs on par with the best llm-based model via more efficient planning and reflection, without requiring manually crafted trace prompts and ad-hoc environment feedback.