C o C o-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation (2024)

Abstract: Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments, especially for graphical user interface (GUI) automation.However, those GUI agents require comprehensive cognition including exhaustive perception and reliable action response.We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP), to systematically improve the GUI automation performance. First, CEP facilitates the GUI perception through different aspects and granularity, including screenshots and complementary detailed layouts for the visual channel and historical actions for the textual channel.Second, CAP decomposes the action prediction into sub-problems: determining the action type and then identifying the action target conditioned on the action type.With our technical design, our agent achieves state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios. Code is available at https://github.com/xbmxb/CoCo-Agent .

## introduction graphical user interface (gui) automation aims to enable human-like operations on operating systems with artificial intelligence instead of human efforts. large language models (llms) have demonstrated commendable performance as human-like agents, showing emergent abilities of perceiving @xcite , reasoning @xcite , and acting @xcite . with the multimodal enhancement, mllm agents * corresponding authors. this research was supported by the joint research project of yangtze river delta science and technology innovation community (no. 2022csjgg1400), the joint funds of the national natural science foundation of china (grant no. u21b2020). current vital challenges for autonomous agents lie in two aspects. one is (i) the dependence on strong (m)llms, and the other is (ii) the insufficient gui environment modeling. although strong (m)llms like gpt-4v (ope-nai, 2023) and chatgpt @xcite ignite the development of autonomous agents, they exhibit shortcomings in realistic use. first, the alignment requires a trustworthy domain transfer as there is a large disparity between gui commands and natural languages. gui agents are expected to generate accurate and well-formulated responses as executable gui commands, which is non-trivial for zero/few-shot prompting. for example, given the gui that accepts commands as "{action: click, touch_point:[y 0 , x 0 ], touch_point:[y 1 , x 1 ], typed_text: ''}", semantically equivalent generations like "open the address book on your phone" is plausible but unavailable. second, the black-box apis are likely to cause unexpected safety issues. risks to privacy and integrity may arise when granting personal device authority to a black-box api. this significantly reduces realistic usability. third, agent performance relies on the prompt design. the issues mentioned above leave heavy burdens on the design of prompt lines for those agents. besides necessary environment descriptions, the prompts (and post-processing) must be sophisticated for domain alignment, instruction following, and security risk mitigation in different circumstances. second, gui agents necessitate a comprehensive multimodal perception for the informative gui environment modeling. existing methods for visual language models are mainly endowed with favorable abilities in semantic alignment between the 1 9097 vision and language modalities @xcite @xcite . however, gui contains fine-grained details and intricate semantic connections, presenting a challenge for agents to comprehend @xcite . consider a screenshot that includes a magnifier icon, where the conventionally accepted meaning of "search" is conveyed. it implies a potential action through implicit semantics despite its small pixel size. thus, only leveraging general image semantics like captioning is insufficient for gui environment modeling. in addition, the gui environmental perception is limited by the finite input window, where a balance between the visual and textual feature length needs to be struck. this work proposes coco-agent, a comprehensive cognitive mllm agent, to address the challenges above for smartphone gui automation. coco-agent adopts a multimodal backbone of llava @xcite and further enhances comprehensive cognition, respectively for exhaustive perception and reliable action response. the two proposed approaches are comprehensive environment perception (cep) and conditional action prediction (cap). specifically, cep integrates gui perception elements of textual goal, historical action, and high-level and detailed description of the vision channel. cap decomposes the complex and redundant gui action commands into sub-problems following a top-down order. our experiments cover diverse tasks in two gui benchmarks, aitw @xcite and meta-gui @xcite , including application manipulation, web operation, and dialogues. coco-agent achieves sota performance with a limited parameter size. subsequently, we present deep analyses including element ablation, visual module selection, and future action prediction. we show the significant effect of each perception element and the favorable choice of visual module. we also analyze the limitations of existing datasets and illustrate the additional potential of coco-agent for realistic scenarios. our contributions are summarized as follows: ## related work this section introduces studies on autonomous language agents and multimodal perception of llms. ## methodology in this section, we will first formulate the gui automation task and then propose our coco-agent. concretely, we will describe our technical designs of cognition, namely, comprehensive environment perception (cep) and conditional action prediction (cap), to improve the gui automation performance systematically. figure 1 shows an overall illustration. ## 9099 action type touch_point lift_point typed_text redefined actions in cap press_home "[-1.0, -1.0]" "[-1.0, -1.0]" "" i need @xmath0.0, -1.0]" "[-1.0, -1.0]" "" i need @xmath1.0, -1.0]" "[-1.0, -1.0]" "" i need @xmath2.0, -1.0]" "[-1.0, -1.0]" "" for this goal, no more action is needed, @xmath3.0, -1.0]" "[-1.0, -1.0]" "{string}" i need @xmath4, "typed_text": "{string}" dual_point "{coordinate}" "{coordinate}" "{string}" i need @xmath5, the location of {item name} is "tap_point": "{coordinate}" dual_point "{coordinate}" "{coordinate}" "{string}" i need @xmath6, the location is "tap_point": "{coordinate}" ## experiments this section will introduce the experimental settings including the dataset, implementation details, and baselines, followed by our empirical results. ## analysis this section presents further analysis and discussions. section 5.1 shows the effects of comprehensive cognition elements by ablation and replace-6 9102 ment. section 5.2 discusses the capability of visual modules, followed by future action prediction in section 5.3. dataset features are analyzed in section 5.4, including action type categories and human evaluation for realistic scenarios. ## 6 conclusion this paper proposes coco-agent, an mllmbased autonomous agent for smartphone gui. our method facilitates comprehensive cognition with exhaustive perception and reliable action response. coco-agent is enhanced with two approaches. comprehensive environment perception (cep) integrates gui perception, and conditional action prediction (cap) enables a decomposition of complex commands. coco-agent achieves sota performance in experiments on two gui datasets. further analysis shows that the agent learns the behavior patterns in smartphone gui. the significance of the proposed enhancements is also verified. we discuss the unbalanced category distributions and the underestimation of agent performance. our work reveals the promising capabilities of mllms as autonomous agents, especially for complex environment perception and action response.