M obile VLM : A Vision-Language Model for Better Intra- and Inter- UI Understanding (2024)

Abstract: Recently, mobile AI agents based on VLMs have been gaining increasing attention. These works typically utilize VLM as a foundation, fine-tuning it with instruction-based mobile datasets. However, these VLMs are typically pre-trained on general-domain data, which often results in a lack of fundamental capabilities specific to the mobile domain. Therefore, they may struggle to recognize specific UI elements and understand intra-UI fine-grained information. In addition, the current fine-tuning task focuses on interacting with the most relevant element for the given instruction. These fine-tuned VLMs may still ignore the relationships between UI pages, neglect the roles of elements in page transitions and lack inter-UI understanding. To address issues, we propose a VLM called MobileVLM, which includes two additional pre-training stages to enhance both intra- and inter-UI understanding. We defined four UI-based pre-training tasks, enabling the model to better perceive fine-grained elements and capture page transition actions. To address the lack of mobile pre-training data, we built a large Chinese mobile dataset Mobile3M from scratch, which contains 3 million UI pages, and real-world transition actions, forming a directed graph structure. Experimental results show MobileVLM excels on both our test set and public mobile benchmarks, outperforming existing VLMs.

## introduction mobile phones are widely used in daily life, and ai agents on mobile platforms are gaining industry and academic attention @xcite . due to the limitations of purely text-based llms in understanding user interface (ui) elements and page structures @xcite , recently released mobile ai agents are mainly driven by vision-language models (vlm) @xcite @xcite . these works typically use vlm as a base model and then fine-tune it on instruction-based mobile datasets for domain adaptation. as a result, they excel in page navigation and can provide a coarse summary of ui functionality. however, these vlms like gpt-4v (openai, 2023), cogvlm @xcite , and qwen-vl @xcite typically utilize large-scale general datasets, such as laion-5b @xcite , coyo @xcite , for pretraining. the proportion of mobile ui pages in these datasets is very low, which results in the overall image characteristics of the datasets being quite different from those of mobile-specific datasets. moreover, the general pre-training task does not particularly consider the characteristics of mobile ui for these vlms. the general pretraining task, such as image caption and visual question answering, mainly focuses on the overall to address these issues, as shown in figure 1 , we propose two additional mobile pre-training stages and four specific mobile tasks to enhance both intra-and inter-ui understanding. in stage 1, 3 ui tasks are implemented to enhance the model's granular understanding of intra-ui content. in stage 2, action prediction tasks are introduced to predict actions connecting two pages, thereby enhancing inter-ui understanding. based on this training framework, we propose mobilevlm, which utilizes consistent mobile data from mobile3m for both pre-training and fine-tuning. this is a vlm that can simultaneously understand fine-grained element information within a ui page and the transition relationships between ui pages. to address the lack of mobile pre-training data, we created mobile3m, a large-scale dataset focusing on third-party chinese apps. specifically, we selected 49 popular apps and iteratively interacted with each ui element, collecting interaction traces. as shown in figure 2 , all interaction traces of each app are combined into a directed graph, where each node represents a ui page and each edge represents a transition action. eventually, mobile3m contains millions of ui pages, xml documents, and page changes caused by user interactions. overall, our work has four major contributions as follows: ## mobile3m dataset in this paper, we propose mobile3m, a large-scale dataset focusing on chinese apps. mobile3m contains a total of 20,138,332 actions, covering 3,098,786 screenshots and corresponding xml documents. these data are organized into 49 large directed graphs, each representing a mobile app, with ui pages as nodes and actions as edges. ## model as shown in ## conclusion we propose mobilevlm, a specialized chinese vision-language model for mobile ui manipulation. it surpasses both open-source mobile vlms and larger closed-source general models on multiple mobile public benchmarks. meanwhile, we build the first large-scale chinese mobile dataset, mo-bile3m, which includes multiple pre-training and fine-tuning tasks specific to mobile scenarios. we hope this work will promote the development of vision-language models in the mobile domain and provide a reference for future mobile-agent research.