C lick A gent: Enhancing UI Location Capabilities of Autonomous Agents (2025)

Abstract: With the growing reliance on digital devices with graphical user interfaces (GUIs) like computers and smartphones, the demand for smart voice assistants has grown significantly. While multimodal large language models (MLLM) like GPT-4V excel in many areas, they struggle with GUI interactions, limiting their effectiveness in automating everyday tasks. In this work, we introduce ClickAgent, a novel framework for building autonomous agents. ClickAgent combines MLLM-driven reasoning and action planning with a separate UI location model that identifies relevant UI elements on the screen. This approach addresses a key limitation of current MLLMs: their inability to accurately locate UI elements. Evaluations conducted using both an Android emulator and a real smartphone show that ClickAgent outperforms other autonomous agents (DigiRL, CogAgent, AppAgent) on the AITW benchmark.

## introduction the current generation of voice assistants (e.g., google assistant, siri, alexa) relies on established spoken language understanding methods such as joint nlu @xcite . however, as these approaches have plateaued, the industry is shifting its focus toward ai agents. these systems, designed to interact with graphical user interfaces (guis) autonomously, are becoming critical for automating tasks on digital devices such as smartphones and computers @xcite . researchers have begun developing agentoriented large language models (llms) @xcite , but the scope of language-only agents is limited in voice assistant applications, where interaction with guis is often needed to perform complex tasks. mllms and visual language models (vlms) offer a promising solution to these limitations @xcite @xcite . unlike language-based agents that rely solely on textual data such as html @xcite or ocr outputs @xcite , mllm-based agents directly interpret visual signals from guis. however, while currentgeneration mllms demonstrate reasonable abilities in screen understanding, reasoning, and action planning, they struggle to locate specific ui elements on screens accurately @xcite . previous works @xcite @xcite attempt to bypass this issue, for example, by using an xml file that details the interactive elements or by using a separate ocr model. yet, such multi-module approaches are error-prone due to the inherent complexity of guis and the inconsistencies in xml/html files. our contribution lies in the development of clickagent, a hybrid autonomous agent that combines mllm-driven reasoning with a specialized ui location model. specifically, clickagent leverages the internvl2.0 mllm @xcite for reasoning and the tinyclick ui location model for identifying the coordinates of relevant ui elements @xcite . this approach significantly improves performance on the aitw benchmark @xcite , surpassing prompt-based agents (d-pot @xcite , cogagent @xcite ) and digirl @xcite , a reinforcement learning-based solution. ## method although models like seeclick @xcite and auto-ui @xcite excel at identifying ui elements, they lack robust action planning, leading to low success rates in real-world smartphone tasks. to overcome these challenges, clickagent integrates internvl2.0 for reasoning, while a dedicated ui location model identifies the exact coordinates of the target ui elements. the inputs to the ui location model are a screenshot and a natural language command corresponding to the desired ui element. clickagent's hybrid approach addresses the limitations of current mllms, which struggle to locate ui elements accurately @xcite . clickagent consists of three main components: decision, ui location, and reflection. in the decision module, the mllm is asked to analyze the current screenshot, review the action history, and determine the next step to complete the user's task. the decision module selects one of the predefined actions: for example, as illustrated in figure 1 , the mllm chooses the click action and issues a ui action command ("click on the eyes closed official video") which, along with a screenshot, serves as input to the ui location model. the model then returns the bounding box coordinates of the relevant ui element. after the action is executed on android, the next screenshot is captured. in the reflection module, the mllm is asked to analyze the screenshot content and the entire action history. reflection evaluates whether the user's task was successfully completed, returning either a "success" or "failure" status. if the task is marked as successful, the autonomous agent is stopped; otherwise, the agent proceeds with the next decision and reflection cycle. internvl2.0 was selected as the primary mllm for the decision and reflection modules due to its strong performance on agentic benchmarks and ease of deployment. seeclick and tinyclick were chosen for ui location tasks because of their high performance on benchmarks such as screenspot @xcite and omni-act @xcite . ## evaluation method we evaluated clickagent on both an android smartphone emulator and a real android smartphone, using the task completion rate (in percentage) as the primary performance metric. this metric assesses whether the agent successfully executed the user's task, making it the most critical measure in autonomous agent evaluation. unlike metrics such as step success rate or action accuracy @xcite @xcite , task completion rate provides a clear assessment of the agent's ability to execute user commands on the device @xcite . our evaluation was performed manually due to the imprecision of current automatic evaluation methods @xcite . we utilized 4 x nvidia a100 80gb gpu for running internvl-2.0 and one nvidia a100 40gb gpu for the ui location models (seeclick, tinyclick). each experiment was repeated three times, showing no significant deviation in accuracy across runs. we report the average of these three runs. except for d-pot @xcite , all results of baseline agents were taken from @xcite . ## main results ## ablation study we conduct an ablation study to understand the impact of two main components (mllm and ui location model) on the overall performance of click-agent. ## conclusion in this paper, we introduced clickagent, a hybrid autonomous agent that combines mllm-driven reasoning with a specialized ui location model. by addressing the limitations of previous approaches in identifying ui elements, clickagent achieves a task completion rate of 73.5 %, outperforming state-of-the-art agents like digirl (69.6 %) on the aitw benchmark. notably, it surpasses existing methods without relying on proprietary mllms such as gemini or gpt-4v. the failures observed in the reflection and decision modules highlight the need for further advancements in mllm capabilities, particularly in understanding uis of less popular apps and websites. one of the most important future directions is research on on-device agents, as maintaining gpu infrastructure is costly, and on-device deployment enables better personalization while preserving user privacy.