R each A gent: Enhancing Mobile Agent via Page Reaching and Operation (2025)

Abstract: Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobileReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the Intersection over Union (IoU) Accuracy and Text Accuracy by 7.12% and 7.69% on the step-level and 4.72% and 4.63% on the task-level compared to the SOTA agent. Our data and code will be released upon acceptance.

## introduction with the quick advancement of visual language models (vlms) @xcite , it has become more feasible to create mobile ai agents that can operate mobile devices @xcite @xcite . some early works @xcite @xcite have attempted to combine powerful general vlm models, such as gpt-4v @xcite , with prompt engineering and retrieval modules to generate ui actions for mobile control. other works have used gui navigation datasets @xcite to fine-tune the base vlms to improve their stability in generating ui actions. subsequent works @xcite figure 1 : an example of a task and its subtasks and possible gui flows. green boxes represent the pages that need to be reached, and orange arrows represent the actions that need to be operated. to complete this task, the agent must reach 5 pages and do 3 operations. although existing mobile ai agents have shown good performance, they still have some limitations. existing agents focus on interacting with the most task-relevant elements of the current page, and ignore whether the entire gui flow solves the task. this method sometimes causes them to focus on single-step action accuracy and select tasksrelated actions greedily, thus falling into a local optimal solution. as shown in figure 2 , for the task of adding xiaomi 14 to the shopping cart, the agent should first reach the product page, but the homepage does not contain elements related to the product name. in this case, it might go directly from the homepage to the shopping cart page. therefore, we break down the task into several subtasks and focus on the agent's subtask completion abilities. intuitively, these subtasks can be divided into two categories, reach and operate. reach only requires the agent to reach a specific page, regardless of the path taken. operate requires the agent to reach a specific page and perform some specific operations. figure 2 shows a task and its gui flow, consisting of a 9-step chain, the key to solving this task is to reach the product page first, and then click the "add to cart" pop-up window and select the attributes and "ok" button in the pop-up window in sequence, as shown in figure 1 . for the product page reach subtask, it can be done either through product search or by browsing the product catalog. similarly, the operate subtask only requires the agent to select the "b" attribute in the pop-up window, and browsing other attributes does not affect the task completion. to address these limitations, we collected a training dataset called mobilereach, which contains three types of tasks: page navigation, page reaching, and page operation. in addition, we propose reachagent, a two-stage framework that focuses on the ability to complete subtasks. in the first stage, in addition to page navigation abilities, it also learns how to reach a specified page and complete operations on a specified page. in the second stage, a 4-level reward function is used to weigh different gui flows and construct preference data to further reinforce reachagent, furthermore, an action alignment mechanism is proposed to reduce the difficulty of action generation. the main contributions of this paper can be summarized as follows: brief task help me find detailed information about xiaomi 14, and add a white one to the shopping cart. (ding, 2024) use prompt engineering methods and rely on existing closed-source models (e.g., gpt-4v, gpt-4o) to achieve mobile control. other studies like cogagent @xcite and agenttuning @xcite use data-driven methods to fine-tune the open-source vlms. in order to improve the device control abilities of mobile ai agents, large-scale datasets with diverse scenarios and accurate annotations are needed. rico @xcite and aitw @xcite are two publicly available large-scale gui datasets, containing 72,219 single-step gui tasks and 715,142 multi-step page navigation tasks, respectively. they are widely used in multiple gui modeling works @xcite @xcite . however, they are constructed by combining crowdsourcing workers and automated annotation, which also leads to noise and wrong labels. to this end, a series of published works clean and filter these datasets to improve their quality. enrico (leiva et al., 2020), ui-bert @xcite , vins @xcite extended the rico dataset and proposed new gui tasks such as ui layout classification and ui element retrieval. autoui @xcite further filtered the googleplay tasks in the aitw dataset. however, these datasets mainly contain screenshots and ocr text, lacking gui raw data such as xml documents, which limits the agent to further obtain mobile environment information. mobilevlm @xcite and @xcite build large-scale pre-training datasets to enhance the agent's ui understanding ability. here, mobile3m @xcite proposed by mobilevlm uses a breadthfirst approach to explore the gui pages of each app and forms these pages into a graph structure. this allows us to obtain environmental information about different paths and operations, for example, the next pages after performing different actions in a page, and how many paths there are from one page to another. however, as a pre-training dataset, mobilevlm lacks fine-tuning tasks. most of their tasks are ui understanding and single-step action generation. a small number of multi-step tasks can be summarized as navigate to a page with a certain button. therefore, based on mobile3m, we extracted available gui flows and annotated a page navigation dataset. using the graph structure environment, we further extracted page reaching and page operation datasets, and scored and paired preference datasets for rl learning. we used the three page datasets for one-stage sft fine-tuning of reachagent and the preference datasets for twostage rl optimization. reinforcement learning (rl). rl-based finetuning methods have shown great potential in improving the generation quality of llms @xcite @xcite . for mobile device control, some agents use rl methods to further optimize fine-tuned models. digrl @xcite uses gemini 1.5 pro @xcite as an automatic evaluator to assign rewards to each gui flow generated by the agent, and updates the model with the annotated gui flows through online rl. distrl @xcite uses multiple workers to collect interaction data and then sends them to a central learner for learning, focusing more on distributed frameworks. these works are limited by data collection efficiency and labor-intensive manual annotation, and therefore cannot be well expanded. our work combines agent generation and graph path sampling to obtain a large number of gui flows, uses page reaching and operation subtasks to set reward rules, and builds preference data for each step of the gui flow. gui page: our algorithm uses an android emulator to run the apps and display the gui pages. a typical gui page consists of a screenshot and an xml document containing multiple elements such as buttons, text boxes, icons, and images. these elements have unique identifiers, such as text, bounds, and resource ids, used to interact with them programmatically. figure 4 shows a gui page example. action: our action set includes four types of actions: click, scroll, input, and complete. figure 3 summarizes the parameters and examples of these actions. when the agent interacts with the ui page, such as clicking a button or entering text, appium 1 sends an action command to the android emulator, which then performs the corresponding action. the changes of the gui page are then captured as a new 1 https://appium.io/docs/en/latest/ screenshot and xml document, which are used as the environment for the next step. ## conclusion in this work, we propose reachagent, a twostage framework that leverages page reaching and page operation tasks to enhance the agent's subtask completion abilities. besides, we use a 4-level reward function to collect preference gui flows, and further enhance the agent's overall task completion ability with reinforcement learning. in addition, we construct a mobile control dataset called mobilereach, which contains 3 categories of tasks: page navigation, page reaching, and page manipulation. experimental results show that reachagent significantly improves iou and text accuracy by 7.12% and 7.69% at the step level and 4.72% and 4.63% at the task level, respectively. our reachagent shows strong task completion abilities by addressing the challenge of mobile ai agents focusing more on single-step action accuracy rather than completing the entire task flow. we hope that mobilereach can serve as a useful resource for breaking down tasks, solving page reach and operation subtasks, and provide assistance for future research.