Unlocking Smarter Device Control: Foresighted Planning with a World Model-Driven Code Execution Approach (2025)

Abstract: The automatic control of mobile devices is essential for efficiently performing complex tasks that involve multiple sequential steps. However, these tasks pose significant challenges due to the limited environmental information available at each step, primarily through visual observations. As a result, current approaches, which typically rely on reactive policies, focus solely on immediate observations and often lead to suboptimal decision-making. To address this problem, we propose Foresighted Planning with World Model-Driven Code Execution (FPWC),a framework that prioritizes natural language understanding and structured reasoning to enhance the agent’s global understanding of the environment by developing a task-oriented, refinable world model at the outset of the task. Foresighted actions are subsequently generated through iterative planning within this world model, executed in the form of executable code. Extensive experiments conducted in simulated environments and on real mobile devices demonstrate that our method outperforms previous approaches, particularly achieving a 44.4% relative improvement in task success rate compared to the state-of-the-art in the simulated environment.

## introduction mobile devices have become central to modern life, facilitating a wide range of activities such as browsing the web, reading news, communicating via email, online chatting, ticket booking, and shopping. as reliance on these devices intensifies, the development of autonomous agents that can seamlessly replicate human interactions to perform such tasks is increasingly crucial. these agents must autonomously interpret textual and visual information from views (i.e., screenshots) and language instructions, navigating the complexity of device interfaces to execute precise actions. recent advancements in large language models (llms) and vision language models (vlms) @xcite provide a promising foundation for developing such agents. when prompted effectively @xcite , these models can generate structured, executable actions given task descriptions and observations. existing works like @xcite and mobile-agent @xcite have leveraged vlms to create device-control agents. however, most existing methods encounter difficulties in making reliable decisions, resulting in a low task success rate. this challenge primarily arises from the complexity of executing tasks on mobile devices, which require a sequence of distinct steps transitioning between views. each step provides only limited contextual information about the underlying digital environment. consequently, to accurately predict globally optimal actions, agents must first develop a comprehensive understanding of the environment. furthermore, they need to implement a forward-looking policy based on this understanding to anticipate the long-term consequences of their actions. in contrast, most prior approaches operate in a reactlike action loop @xcite , heavily relying on information from the current observation and a coarse-grained historical context, which leads to myopic decision-making. to overcome these limitations, we propose foresighted planning with world model-driven code execution (fpwc), a novel framework that enables agents to construct a global understanding of the environment they operate in, referred to as the world model @xcite ) (see figure 1 ). a world model is essentially a simulation of the environment, such that planning can be done directly within the model, and it enables the agent to predict future states and outcomes without direct interaction with the real environment @xcite . we represent the world model as a textbased directed graph, where descriptions of views are the nodes and transitions between views are edges. the agent initializes this model given task description, using pre-trained knowledge in vlms to infer precise, task-relevant contents written as structured texts. the agent then makes explicit plans by specifying a rule-based policy written as executable code, specifying how an agent should traverse the world model in order to complete the task. as tasks progress, the agent dynamically refines the world model and the planning process in a closed loop, ensuring adaptability and online learning. to constrain the complexity of the constructed world model, we strictly limit it to describing a single mobile application that is most pertinent to the task at hand. to adapt our method for tasks necessitating interaction between multiple applications, we incorporate a recursive function within the framework. this function, when called in a plan, generates a sub-agent responsible for executing sub-tasks in other applications that are essential for the current task. the sub-agent mirrors the main agent, possessing its own world models and plans for the sub-task delegated by the main agent, where natural language instructions guide both high-level planning and low-level execution. in summary, this paper makes the following contributions: ## methodology to help the device-control agent make informed and foresighted decisions at each step, we endow it with the capability to generate grounded, executable plans based on a self-constructed and dynamically updated world model -a structured, textbased representation of the app. the following sections provide a detailed overview of our framework fpwc. see figure 1 for an overview. ## experiment through our experiments, we aim to address three pivotal questions that align with the objectives outlined in our study: (1) how does our approach compare to state-of-the-art device-control methods in terms of metrics such as success rate and efficiency across benchmark tasks? (2) can our method effectively tackle tasks that previous approaches fail to solve due to the demand for foresight planning? (3) how well does our agent generalize to tasks on real-world mobile devices, including cross-application scenarios? in order to gain deeper insights, we further evaluate (a) the extent to which individual components contribute to the overall performance of our agent, and (b) the trade-offs between computational efficiency (e.g., latency, resource usage) and task execution performance. ## conclusion in this work, we proposed a novel framework for intelligent device control that integrates graphbased world models to enhance task planning and decision-making. by reformulating planning as a code-generation process, our approach enables the iterative refinement of executable python scripts through a vlm, achieving adaptability and lifelong learning capabilities. to address challenges in spatial localization and fine-grained scene understanding, we introduced a zoom-in self-verification mechanism that improves accuracy in complex environments. experimental results demonstrated significant performance improvements over exist-ing methods in autonomous device control tasks, validating the effectiveness of our framework.