Android in the Zoo: Chain-of-Action-Thought for GUI Agents (2024)

Abstract: Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone, which completes a task triggered by natural language through predicting a sequence of actions of API. Even though the task highly relies on past actions and visual observations, existing studies typically consider little semantic information carried out by intermediate screenshots and screen operations. To address this, this work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. We demonstrate that, in a zero-shot setting upon three off-the-shelf LMMs, CoAT significantly improves the action prediction compared to previous proposed context modeling. To further facilitate the research in this line, we construct a dataset Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs together with chain-of-action-thought annotations. Experiments show that fine-tuning a 1B model (i.e. AUTO-UI-base) on our AitZ dataset achieves on-par performance with CogAgent-Chat-18B.

## introduction nowadays, smartphones have become an essential part of daily lives. autonomous operation of graphical user interfaces (gui) by human instructions can substantially simplify everyday routines. such tasks, formalized as gui navigation @xcite , therefore carry immense social importance, especially for people with physical disabilities @xcite . recent works have explored prompt engineering @xcite , finetuning @xcite and memory augmentation @xcite to utilize the capability of large language models (llm) on interactive mobile environments. however, progress is held back due to the scarcity of attention paid on the underlying semantics of smartphone operations. gui navigation usually entails initially observing the screen, considering the next action to take, and reflecting on the outcome of that action @xcite . previous works @xcite ignore the logic behind diverse actions on the screen, concentrating solely on the coordinates of an operation, such as "click on (0.17, 0.89)", which is quite insufficient. as shown in figure 1 , we need explicit explanations for the intermediate results during gui navigation: in order to equip existing gui agents with such capability, we summarize the series of navigation steps as chain-of-action-thought (coat), including the screen description, the thinking process about the next action, the textual next action description, and the possible action outcomes. screen description, together with the screenshots, provides the agent with information basis for decisionmaking @xcite . whereas action think, action description and action result demonstrate the rationale between operations. equipped with coat, we achieve significant improvements in the show the shopping cart on walmart. add "logitech g pro" to the cart on walmart. ## android in the zoo (aitz) there is a lack of data that captures the underlying semantics of the coat paradigm, hindering small models from obtaining this ability. we therefore propose to construct a novel, high-quality and comprehensive dataset to remedy this blank. ## conclusion in conclusion, our work aims to bolster the navigation ability of lmm-based gui agents. we propose chain-of-action-thought (coat) by analyzing human orienteering processes. we start by verifying that coat is superior to three typical context modeling methods. in order to inject coatlike thinking capabilities into existing gui agents, we further generated a set of high-quality coatdriven data through cooperation between human experts and gpt-4v, namely android-in-the-zoo (aitz) dataset. aitz enriches this field with a robust dataset that bridges perception and cognition, facilitating effective training and reliable evaluation for gui navigation agents. experiments demonstrate the efficiency and usefulness of proposed chain-of-action-thought paradigm.