You Only Look at Screens: Multimodal Chain-of-Action Agents (2024)

Abstract: Autonomous graphical user interface (GUI) agents aim to facilitate task automation by interacting with the user interface without manual intervention. Recent studies have investigated eliciting the capabilities of large language models (LLMs) for effective engagement in diverse environments. To align with the input-output requirement of LLMs, most existing approaches are developed under a sandbox setting where they rely on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. Consequently, those approaches often grapple with inference inefficiency and error propagation risks. To mitigate the challenges, we introduce Auto-GUI, a multimodal solution that directly interacts with the interface, bypassing the need for environment parsing or reliance on application-dependent APIs. Moreover, we propose a chain-of-action technique—leveraging a series of intermediate previous action histories and future action plans—to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30 K unique instructions, spanning multi-step tasks such as application operation, web searching, and web shopping. Experimental results show that Auto-GUI achieves state-of-the-art performance with an action type prediction accuracy of 90% and an overall action success rate of 74%. Code is publicly available at https://github.com/cooelf/Auto-GUI.

## introduction building intelligent autonomous agents that are capable of task planning, decision making, and action execution in a particular environment is a longstanding goal of artificial intelligence (ai) @xcite @xcite . the advent of large language models (llms) @xcite * work done at amazon web services. 2022; openai, 2023) has flourished promising opportunities for developing autonomous agents to assist users in completing tasks in distinct environments such as operation systems, specific applications, and web browsers @xcite @xcite @xcite @xcite . recent studies have explored prompt engineering @xcite @xcite @xcite and fine-tuning techniques @xcite @xcite to elicit the capability of language models to execute actions in interactive environments. however, there are at least two major challenges that have limited real-world applications of autonomous agents. first, existing approaches commonly rely on external tools such as optical character recognition (ocr) and icon detectors @xcite to parse the environment into textual elements (e.g., html layouts) as inputs to a language model (figure 1(a) ) @xcite . on the one hand, the parsed elements generate lengthy inputs, thus leading to inference inefficiency. since computational latency is a key measure in deployment, using lengthy inputs would increase inference cost and may even exceed the input length limit of the language model. on the other hand, parsing the visual environment into textual elements may also be prone to error propagation or information loss because parsing mistakes are inevitable using external tools. second, most existing approaches are under the sand-box setting that requires accessing internal apis to interact with the environment @xcite , e.g., using a javascript element selection on a webpage or a python interpreter to execute actions. however in practice, the api interface is often inaccessible in third-party applications (apps). screen parsing: ocr, icon detection, html conversion language model application-specific api @xmath0.com/search?searcht" >lowes.com/search?search @xmath1; irwin, 1989)-allowing direct interactions on the screen without needing access to intermediate environment parsing or interval application-dependent apis (figure 1(b) ). to address the challenges, we introduce auto-gui, a multimodal approach that directly interacts with the graphical user interfaces (guis). to further strengthen the agent's action prediction capability, we propose a novel chain-of-action technique, where a chain of action is a series of intermediate previous action histories and future action plans. we evaluate auto-gui on a new device-control benchmark aitw @xcite with 30k unique instructions, spanning multi-step tasks of application operation, web searching, and web shopping. experimental results show that auto-gui achieves state-of-the-art performance with an action type prediction accuracy of 90% and an action success rate of 74%. in summary, our work makes the following technical contributions: our work falls into the field of language agents. this section will first review the recent progress in building language agents and then discuss the approaches to conduct user interface control with language agents. ## methodology in this section, we will first introduce the basic concepts for the gui control task and then describe the design of our proposed auto-gui framework. ## conclusion this work presents an autonomous gui agent called auto-gui that can interact in a multimodal gui environment without environment parsing or application-dependent api access. in addition, we propose a chain-of-action technique that leverages the previously executed actions and future action plans to help the agent decide what action to execute. experimental results show that auto-gui achieves superior performance to previous prompting-based and fine-tuning baselines. we show that it is possible to achieve state-of-the-art performance by an end-to-end model without relying on external tools and application-specific apis to parse the environment and interpret the predicted actions. besides the strong performance and generality across domains, auto-gui is able to infer an action in less than one second.