M ob A : Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation (2025)

Abstract: Existing Multimodal Large Language Model (MLLM)-based agents face significant challenges in handling complex GUI (Graphical User Interface) interactions on devices. These challenges arise from the dynamic and structured nature of GUI environments, which integrate text, images, and spatial relationships, as well as the variability in action spaces across different pages and tasks. To address these limitations, we propose MobA, a novel MLLM-based mobile assistant system. MobA introduces an adaptive planning module that incorporates a reflection mechanism for error recovery and dynamically adjusts plans to align with the real environment contexts and action module’s execution capacity. Additionally, a multifaceted memory module provides comprehensive memory support to enhance adaptability and efficiency. We also present MobBench, a dataset designed for complex mobile interactions. Experimental results on MobBench and AndroidArena demonstrate MobA’s ability to handle dynamic GUI environments and perform complex mobile tasks.

## introduction multimodal large language models (mllms) have seen significant advancements in recent years, supported by vast multimodal datasets. these models @xcite @xcite @xcite @xcite @xcite @xcite @xcite excel in tasks such as chain-of-thought (cot) reasoning @xcite , in-context learning (icl) @xcite , and various applications (wang et al., 2024b; wang and zhao, 2023; chen et al., 2024a; liu et al., 2024b; pan * corresponding authors are lu @xcite @xcite @xcite . their capabilities have also enabled new mllm-based agents for real-world tasks @xcite @xcite @xcite @xcite @xcite @xcite . however, mllms face significant challenges when addressing complex gui interactions and facing diverse user demands in real-world scenarios, particularly on devices such as smartphones @xcite and computers @xcite . on the one hand, gui environments are highly diverse and pose different action spaces across different apps and pages. for instance, the number and position of clickable icons can vary greatly across pages; some pages require text input, while others involve scrollable elements. such variability makes proactive task planning hardly adapt to the real environment contexts and thus become infeasible to complete. on the other hand, the action executor can also lack capabilities enough to achieve it, even given a feasible task plan. in all these cases, agents with trivial or static planning @xcite @xcite @xcite will fail to align with the environment contexts and action executor's capacity and thus can fail the whole task easily caused by failure of a single sub-task. furthermore, existing mllm-based gui agents @xcite @xcite often lack a powerful and comprehensive memory to face the need for dynamic planning at various levels and diverse user demands. these problems hinder the design of a practical mobile assistant. to address these challenges, we propose moba, a novel mllm-based mobile assistant system with an adaptive planning module that dynamically adjusts task plans according to the execution results. as proactive planning often fails to accurately determine the actions required in a specific application or page or to align with the action executor's capacity, moba leverages reflection mechanism to recover task execution from failed sub-plans by reassessing goals or breaking tasks into more fine-grained sub-goals. to better support adaptive planning with various sub-goal granularity, a multifaceted memory module providing hierarchical memory support is proposed. we also introduce mobbench, a diverse dataset for complex mobile interactions, and demonstrate moba's effectiveness on mobbench and androidarena @xcite , showing its capability to handle dynamic gui environments. our contributions are threefold: ## the moba system the system overview of moba is shown in figplan module interprets the user's command (➀) and resolves the task into several easier and clearer sub-tasks adaptively with the help of experiences in the multifaceted memory module (➁), while the reflection module will review if the decomposition is feasible and sub-goals are achievable. then, under the direction of a specific sub-goal (➂), the local agent will leverage the experiences in the memory module (➃), predict the concrete actions, and directly control the device (➄). after la's execution, the reflection module will reflect if the current sub-task has been completed (➅) and the plan module can revise the plan accordingly. the memory module can also be updated after the invocation of the plan module and local agent (➆) to improve moba's performance through execution. to ameliorate the performance at the early stage of the memory, it can also be initialized with a warm-up of some basic expert experiences. finally, moba can generate a response to the user regarding the result of task execution (➇). the remaining parts of this section will elaborate on the proposed adaptive plan module and multifaceted memory module. ## experiments to comprehensively compare moba with other gui agents in handling complex user instructions and executing gui interactions on mobile devices, we evaluate them using a real-life scenario test set called mobbench. additionally, we assess our method using a widely adopted mobile benchmark, android arena. ## case study figure 3 demonstrates how the adaptive planning and multifaceted memory support task completion in moba. moba can accurately interpret user intent from command "help me check when will i reach the travel destination tomorrow." and give decomposed sub-tasks based on historical commands. for sub-task 1, moba retrieves relevant details from app and user memory, extracts key information (train schedule and destination), and stores it in action memory. when encountering failures, moba uses historical experiences to reflect and adapt. during sub-task 3, when moba initially failed to input the train number using the box_input function, it reflects on its previous operations and employs a character-by-character input method, completing the task. the key feature of this page will be saved into page memory, thus moba is unlikely to encounter the same failure. additionally, memory retrieval is crucial for handling contextual tasks. in sub-tasks 9 and 13, al-though the user doesn't explicitly specify the travel date or destination in the task request. moba can rely on previously stored action memory data to provide an accurate response. ## conclusion and future works this paper presented moba, an innovative mobile phone assistant system empowered by mllms. utilizing a two-level agent structure, comprising a global agent and a local agent, moba effectively understands user commands, plans tasks, and executes actions. the combination of memory and plan modules enhances its ability to learn from previous interactions, improving efficiency and accuracy. our evaluations demonstrated that moba surpasses existing mobile assistants in handling complex tasks, leveraging multi-level memory, task decomposition, and action-validation mechanisms. these features enable precise task execution even with intricate or indirect commands. future work will focus on improving the performance on imageonly scenarios where the view hierarchy is unattainable, deploying an end-side model on mobile phones for faster response and secured privacy. we will continue to expand mobbench by adding more popular applications from different regions and languages. we hope moba illustrates the potential of mllms-empowered mobile assistants and provides valuable insights for future works. useful: <element @xmath0.view.viewgroup" @xmath1,605][947,770]" @xmath2: <node @xmath3.widget.framelayou t" @xmath4.mobileticket" @xmath5,0][1440,3120]">