INREACT : An Inspire-Then-Reinforce Training Framework For Multimodal GUI Agent (2025)

Abstract: Graphical User Interface (GUI) interaction, which aims to develop an intelligent GUI agent that executes user instructions to perform tasks such as installing applications by controlling digital devices, has gained significant attention due to its practical value. Although current advanced multimodal large language models (LLMs) provide GUI agents with robust perception and reasoning capabilities, they often struggle with the precise localization of small elements. To tackle this problem, we propose InReAct, a multimodal GUI agent framework that unifies observing, thinking, and acting for precise and interpretable decision-making. It is trained via a two-stage process: curriculum learning to progressively build perception, grounding, and reasoning abilities, followed by reinforcement learning to refine pixel-level grounding with an outcome-based reward. We introduce a rule-based reward function that jointly optimizes action-type selection and pixel-level localization accuracy. Experimental results on multiple datasets demonstrate the superiority of InReAct in both grounding and navigation tasks.

## introduction graphical user interface (gui) interaction refers to a class of sequential decision-making tasks in which agents interpret user instructions and manipulate digital interfaces, including pcs and mobile devices, to accomplish specific goals. these interactions typically involve executing low-level operations like click, type, or scroll in the correct order to complete complex tasks, such as searching for information. a capable gui agent must possess two core abilities: visual observation and reasoning. observation is particularly critical in gui environments like pcs and mobile devices, where visual * equal contribution † corresponding author components are often densely packed and visually similar, making precise localization challenging. meanwhile, reasoning is essential for complex, goal-directed interactions that involve sequential decision-making. unlike single-step tasks, gui interaction requires agents to consider both interface states and past actions. with the emergence of powerful multimodal large language models (mllms) @xcite @xcite , recent approaches have enabled screenshot-based gui control without relying on structured metadata such as html or accessibility trees @xcite . however, despite these advances, existing gui agents still struggle with accurate grounding and coherent reasoning in complex, real-world scenarios. to address the challenges of imprecise grounding and weak multi-step reasoning, recent efforts have focused on training mllm-based gui agents via supervised fine-tuning (sft) or reinforcement learning (rl). in sft, models are trained on paired gui-action datasets to align visual observations with user instructions. some approaches @xcite @xcite enhance sft by incorporating instruction tuning, vision-language pretraining, or synthetic data generation to boost coverage. however, these strategies commonly adopt a flat optimization objective, lacking curriculum or structural guidance, which limits the model's ability to progressively acquire complex capabilities. in contrast, reinforcement learning introduces outcome-driven optimization through environment feedback. some methods @xcite rely on sparse success signals to guide multi-step task completion. despite this, many rl strategies remain coarse and fail to capture pixel-level grounding precision. moreover, most rl pipelines lack strong initialization or structured learning stages, often resulting in unstable policies and inconsistent behaviors. in this work, we propose inreact, a structured training framework for gui agents that combines curriculum learning with reinforcement learning. the curriculum learning stage is designed to systematically build core agent capabilities-including perception, grounding, and reasoning through a staged learning process of increasing complexity. furthermore, we introduce a reinforcement learning strategy with rule-based reward functions that jointly considers action-type correctness and distance-aware spatial accuracy, enabling more precise and reliable interactions in complex gui environments. our contributions are summarized as follows: • we propose a novel multimodal gui agent framework, called inreact, which synergizes observing, thinking, and acting to make more interpretable and goal-directed decisions when handling complex gui navigation tasks. ## conclusion in this paper, we present inreact, a multimodal gui agent framework that integrates observation, reasoning, and action for complex gui interactions. it is trained in two stages: curriculum-learning to build perception, grounding, and reasoning skills, followed by reinforcement learning with a taskspecific reward that improves action correctness and spatial precision. we evaluate inreact across four datasets: screenspot, screenspot-pro, mind2web, and an-droidcontrol, and demonstrate consistent improvements over prior methods. our findings highlight the effectiveness of combining structured learning and outcome-driven optimization for enhancing gui agent performance, setting a new direction for generalizable and interpretable vision-language agents.