M obile A 3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users (2025)

Abstract: The advancement of mobile GUI agents has opened new opportunities for automating tasks on mobile devices. Training these agents requires large-scale high-quality data, which is prohibitively expensive when relying on human labor. Given the vast population of global mobile phone users, if automated data collection from them becomes feasible, the resulting data volume and the subsequently trained mobile agents could reach unprecedented levels. Nevertheless, two major challenges arise: (1) extracting user instructions without human intervention and (2) utilizing distributed user data while preserving privacy.To tackle these challenges, we propose MobileA3gent, a collaborative framework that trains mobile GUI Agents using decentralized self-sourced data from diverse users. The framework comprises two components, each targeting a specific challenge: (1) Auto-Annotation, which enables the automatic collection of high-quality datasets during users’ routine phone usage with minimal cost. (2) FedVLM-A, which enhances federated VLM training under non-IID distributions by incorporating adapted global aggregation based on both episode-level and step-level variability. Extensive experiments prove that MobileA3gent achieves superior performance over traditional approaches at only 1% of the cost, highlighting its potential for real-world applications. Our code is publicly available at: https://anonymous.4open.science/r/MobileA3gent-Anonymous.

## introduction mobile gui agents @xcite have experienced significant advancements, propelled by recent progress in vision-language models (vlms). designed to simulate human mobile phone usage behavior, mobile agents can automate complex tasks on mobile phones, saving tremendous human labor and change everyday lives. compared to non-agent the training of mobile agents heavily depends on largescale, high-quality datasets @xcite . to build such datasets, existing approaches rely on centralized data collection followed by human annotation, resulting in high costs and limited scalability. to achieve large-scale data acquisition more efficiently, a paradigm shift (as shown in figure 1 ) from centralized to distributed data collection is necessary, enabling diverse users to participate in data contribution. additionally, replacing human annotation with automatic annotation is crucial for efficiently processing the vast amount of collected data, allowing direct data sourcing from real user interactions. our insight is that the frequent and ever-growing phone usage by users worldwide naturally generates valuable supervisory information, which can serve as a rich data source for training mobile agents. building on this user-centric insight, we aim to effectively utilize these distributed data, while minimizing human involvement in the process. however, two technical challenges remain: 1. although the users' phone usage provides realworld trajectories (screenshots and actions), it is difficult to extract the real intentions (instructions) behind the actions in natural language; 2. data collected from one user is both scale-limited and privacy-sensitive. the challenge lies in how to utilize distributed data from diverse users to boost performance while protecting privacy. to tackle these challenges, we propose mo-bilea3gent, a collaborative learning framework that trains mobile agents using automatically collected user data from daily phone interactions while preserving user privacy. specifically, mo-bilea3gent features two novel techniques. first, we propose auto-annotation, an automated method for data collection and annotation that leverages locally deployed vlms to annotate user instructions based on interaction trajectories. the key technical innovation lies in combining step-wise low-level instruction breakdowns with episode-wise summarization, allowing even small local vlms to better understand the user's intent. the step-wise description decomposes complex user instructions into simpler steps, enabling the vlm to comprehend and extract information more accurately. meanwhile, the episode-wise summarization provides a global perspective on the entire task, generating a more comprehensive caption of the user's ultimate instruction. compared with human annotation, auto-annotation generates data of comparable quality with minimal cost requirement. second, to effectively utilize decentralized data from diverse users, we propose fedvlm-a, which pioneers the integration of federated learning (fl) @xcite and collaborative training of vlm-based gui agents, while ensuring rigorous user privacy protection. we further propose a novel aggregation method, termed adapted global aggregation, which accounts for both episode-level and step-level distributions to handle the two-level heterogeneity (formulated in section b) in diverse users' data, overcoming the limitations of traditional one-level aggregation methods @xcite @xcite . adapted aggregation adapts the global aggregation weights using a weighted sum of episode and step counts for each client, thereby enhancing the performance of mobile agents trained in non-iid scenarios. extensive experiments on four benchmarks with 10+ models and metrics demonstrate that: (1) mo-bilea3gent achieves the best all-around trade-off across four dimensions, delivering performance on par with centralized manual approaches at significantly lower cost, while also ensuring privacy and achieving exceptional scalability. (2) auto-annotation outperforms all annotation baselines in performance while reducing annotation costs by 99% compared to manual labeling. (3) fedvlm-a achieves an at least 5% relative improvement over representative fl baselines in non-iid scenarios. these promising results underscore the immense potential of our framework to serve as a novel and practical paradigm for real-world applications. to summarize, our contributions are as follows: 1. we formulate the problem of self-sourced data collection from distributed mobile phone users and propose auto-annotation, an automatic data collection method, which achieves data quality comparable to human-annotated data at a significantly lower cost. 2. we introduce mobilea3gent, a collaborative framework for training mobile agents on decentralized user data while preserving privacy. by incorporating fedvlm-a, we enable federated training of vlms and achieve superior performance when confronted with heterogeneity. 3. we conduct extensive experiments across comprehensive benchmarks and metrics. the compelling results highlight the substantial potential of our approach for real-world applications. ## conclusion to overcome the scalability and efficiency limitations of traditional mobile agent paradigm, we emphasize the necessity of transitioning from centralized to distributed user-centric data collection, and from human to automatic annotation. to achieve this, we propose mobilea3gent, a framework that collaboratively trains mobile gui agents using selfsourced data from diverse users. specifically, we introduce auto-annotation, an efficient approach for generating high-quality datasets from routine phone usage at minimal cost. additionally, we present fedvlm-a, a federated vlm training framework with adapted global aggregation to handle mobile data heterogeneity. extensive experiments on four benchmarks with 10+ models and metrics validate the effectiveness of mobilea3gent. the promising results highlight the scalability and practicality of our user-centric paradigm, offering a privacy-preserving and cost-efficient solution for training large-scale mobile agent.