GUI Agents: A Survey (2025)

Abstract: Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.

## introduction large foundation models (lfms) have significantly transformed both the landscape of ai research and day-to-day life @xcite @xcite @xcite . recently, we have witnessed a paradigm shift from using lfms purely as conversational chatbots @xcite @xcite to employing them for performing actions and automating useful tasks (wang et al., 2024b; zhao et al., 2023; yao et al., 2023; shinn et al., 2023; shen * corresponding author: dangmn@umd.edu @xcite . in this direction, one approach stands out: leveraging lfms to interact with digital systems, such as desktops and mobile phones, or software applications such as a web browser, through graphical user interfaces (guis) in the same way humans do, for example, by controlling the mouse and keyboard to interact with visual elements displayed on a device's monitor @xcite @xcite . this approach holds great potential, as guis are ubiquitous across almost all computer devices that humans interact with in their work and daily lives. however, deploying lfms in such environments poses unique challenges, such as dynamic layouts, diverse graphical designs across different platforms, and grounding issues, for instance, fine-grained recognition of elements within a page that are often small, numerous, and scattered @xcite . despite these challenges, many early efforts have shown significant promise @xcite , and growing interest from major players in the field is becoming evident 1 . given the immense potential and rapid progress in this field, we propose a unified and systematic framework to categorize the various types of contributions within this space. organization of this survey. we begin our survey by clearly defining the term "gui agent," followed by a formal definition of gui agent tasks in section 2. we then summarize different datasets and environments in section 3 to provide readers a clearer picture of the kinds of problem settings currently available. we summarize various gui agent architectural designs in section 4, followed by different ways of training them in section 5. lastly, we discuss open problems and future prospects of gui agent research in section 6. ## preliminaries this section formally defines the term "gui agent" and presents a formalization of gui agent tasks. definition 1 (gui agent). an intelligent autonomous agent that interacts with digital platforms, such as desktops, or mobile phones, through their graphical user interface. it identifies and observes interactable visual elements displayed on the device's screen and engages with them by clicking, typing, or tapping, mimicking the interaction patterns of a human user. problem formulation. gui agent tasks involve an agent interacting with an environment in a sequential manner. the environment can generally be modeled as a partially observable markov decision process (pomdp) @xcite , defined by a tuple (u, a, s, o, t ), where u is the task space, a is the action space, s is the state space (not fully observable to the agent), o is the observation space, and t : s × @xmath0. a gui agent is a policy π : ∆ @xmath1, where ∆ s denotes the probability simplex over the state states. most commonly, this is implemented using the entire history of past actions and observations. given a task @xmath2, the agent proceeds through a sequence of actions to complete the task. at each step t, based on the history of past actions and observations, the policy π selects the next action @xmath3. the environment then transitions to a new state s ′ ∈ s according to t . depending on the environment's design, the agent may receive a reward @xmath4, a, s ′ ), where r : s × a × @xmath5. when evaluating gui agents, it is crucial to distinguish between an environment and a dataset. a dataset is a static collection of data point, where each consists of several input features (e.g., a question, a screenshot of the environment, or the current state of the environment) and some output features (e.g., correct answers or actions to be taken). a dataset remains unchanged throughout the evaluation process. in contrast, an environment is an interactive simulation that represents a real-world scenario of interest. a gui environment includes the gui interface of a mobile phone or a desktop. unlike datasets, environments are dynamic, actions taken within the environment can alter its state, hence, allowing modeling the problem as markov decision processes (mdps) or partially observable mdps (pomdps), with defined action, state, and observation spaces, and a state transition function. another critical dimension of the existing benchmarks for gui agents is the distinction between the open-world and closed-world assumptions. closedworld datasets or environments presume that all necessary knowledge for solving a task is contained within the benchmark itself. in contrast, open-world benchmarks relax this constraint, allowing relevant information required to complete a task to exist outside the benchmark. we present a summary of existing gui agent benchmarks in ## gui agent architectures this section focuses on various architectural designs of a gui agent, which we categorize into four main types: (1) perception: designs that enable the gui agent to perceive and interpret observations from its environment; (2) reasoning: designs related to the cognitive processes of a gui agent, such as using an external knowledge base for long-term memory access or a world model of the environment to support other modules like planning; ## gui agent training methods this section summarizes different strategies to elicit the ability to solve agentic tasks in a gui agent agent. we broadly categorize these strategies into two types: (1) prompt-based methods and (2) training-based methods. prompt-based methods do not involve the training of parameters; they elicit the ability to solve agentic tasks by providing detailed instructions within the prompt. training-based methods, on the other hand, involve optimizing the agent's parameters to maximize an objective, such as pretraining, fine-tuning, or reinforcement learning. we present a taxonomy of gui agent training methods in figure 2 . ## open problems & challenges user intent understanding. gui agents still struggle to accurately infer user goals across diverse applications, achieving only 51.1% accuracy on unseen websites @xcite . designing models that generalize effectively across varying tasks is crucial, particularly for handling contextual variations in user interactions @xcite and predicting user behavior in complex interfaces @xcite . a prospective future research direction is to leverage robust training techniques to enable agents to adapt to new environments with minimal retraining, ultimately providing more seamless and adaptive user experiences. other promising directions could include training on diverse user interaction datasets and incorporating context-aware learning techniques that utilize historical user actions to better predict intent. ## conclusion in this survey, we have thoroughly explored gui agents, examining various benchmarks, agent architectures, and training methods. although considerable strides have been made, problems such as intent understanding, security, latency, and personalization remain critical challenges. we hope that this survey is a valuable resource for researchers, offering structure and practical guidance in this rapidly growing and exciting field, and inspiring more work on gui agents. the progress in this area has already benefited mankind, enhancing our daily productivity and transforming the way we interact with computers.