Dynamic Planning for LLM -based Graphical User Interface Automation (2024)

Abstract: The advent of large language models (LLMs) has spurred considerable interest in advancing autonomous LLMs-based agents, particularly in intriguing applications within smartphone graphical user interfaces (GUIs). When presented with a task goal, these agents typically emulate human actions within a GUI environment until the task is completed. However, a key challenge lies in devising effective plans to guide action prediction in GUI tasks, though planning have been widely recognized as effective for decomposing complex tasks into a series of steps. Specifically, given the dynamic nature of environmental GUIs following action execution, it is crucial to dynamically adapt plans based on environmental feedback and action history.We show that the widely-used ReAct approach fails due to the excessively long historical dialogues. To address this challenge, we propose a novel approach called Dynamic Planning of Thoughts (D-PoT) for LLM-based GUI agents.D-PoT involves the dynamic adjustment of planning based on the environmental feedback and execution history. Experimental results reveal that the proposed D-PoT significantly surpassed the strong GPT-4V baseline by +12.7% (34.66% \rightarrow 47.36%) in accuracy. The analysis highlights the generality of dynamic planning in different backbone LLMs, as well as the benefits in mitigating hallucinations and adapting to unseen tasks. Code is available at https://github.com/sqzhang-lazy/D-PoT.

## introduction building autonomous agents capable of assisting humans in addressing real-world challenges has long been a central pursuit of artificial intelligence research @xcite @xcite . recently, there has been a surge in exploration within * corresponding authors the realm of autonomous agents, fueled largely by the emergence of large language models (llms) @xcite @xcite . one prevalent application scenario involves automating graphical user interfaces (guis) on smartphones, where llms are tasked with perceiving smartphone guis and sequentially predicting action commands until the task is completed @xcite . while previous studies have made significant strides by enhancing environment perception through fine-grained gui grounding @xcite @xcite @xcite @xcite , there has been limited focus on the planning capabilities of gui agents. evidence suggests that decomposing a complex task into a series of plans is effective in eliciting the ability of llms within agent systems @xcite @xcite . additionally, given that the environment evolves based on action predictions, it is crucial to dynamically adapt plans based on environmental feedback and execution history. existing llms-based gui agents typically takes actions directly prior planning or adjustment of plans based on environmental feedback (e.g., new gui screenshot) and execution history (e.g., previous steps described in natural language). for instance, as depicted in figure 1 , the static foot_0 methods (black data stream) directly predicts actions based on the screenshot and goal. those approaches struggles to handle complex real-world scenarios, where users often adjust subsequent actions based on past steps. we will show that the widely-used react approach @xcite fails due to excessively long historical dialogues, revealing its inadequacy in handling complex realworld scenarios (section 3). to address the challenge above, we propose a novel method called dynamic planning of thoughts (d-pot) method to enable the llmbased agent to formulate effective plans based on environment feedback and execution history during task execution (with dashed lines in figure 1 ). concretely, d-pot dynamically adjusts its plans by incorporating new screenshots and execution history throughout the goal attainment process. moreover, our proposed method allows for continuous refinement of the current plan, ensuring persistent optimization until the desired goal is achieved. experimental results demonstrate that our planning mechanism substantially improves the task performance. additionally, analysis highlights the efficacy of dynamic planning in mitigating hallucinations and adapting to unseen tasks. our key contributions are as follows: ## method in light of the above experimental results and analysis, we propose dynamic planning of thoughts (d-pot) to mitigate the challenge. on a high level, d-pot consists of three stages: ## conclusion this study introduces a prompting approach called d-pot, designed to facilitate interactions in a multimodal environment. d-pot encourages llms to dynamically update planning based on feedback from the environment and execution history. through the application of d-pot, we demonstrate that the d-pot surpasses the widely adopted gpt-4v baseline on the aitw benchmark dataset. meanwhile, our findings indicate that the d-pot excels in adapting to unfamiliar tasks, and can predict different actions more correctly.