UFO : A UI -Focused Agent for Windows OS Interaction (2025)

Abstract: We introduce UFO, a UI-Fcused agent designed to fulfill user requests tailored to Windows OS applications by observing and analyzing the GUI and control information of these applications. UFO utilizes a hierarchical dual-agent framework that decomposes user requests using a divide-and-conquer approach, enabling seamless navigation and addressing sub-tasks across multiple applications. It also incorporates a control interaction module tailored for Windows OS, which detects control elements effectively and allows for fully automated execution. As a result, UFO simplifies complex and time-consuming processes into tasks that can be completed with natural language commands.We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios. The results derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFOin fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS.

## introduction the application of visual large language models (vlms) has demonstrated transformative potential in addressing complex multimodal challenges, emulating human-like reasoning, planning, and collaboration @xcite @xcite . graphical user interfaces (guis), though designed primarily for human interaction, offer a critical interface through which vlms can engage with software applications. by perceiving and analyzing visual information from ui elements, vlms can ground their actions in the real world, bridging the gap * corresponding author. in this context, the windows operating system (os) emerges as a representative platform due to its frequent use in daily life @xcite , the variety of versatile applications built upon it @xcite , and the complexity of tasks that often require long-term planning and interaction across multiple applications @xcite . additionally, the system's emphasis on gui interaction highlights the need for a general intelligent agent capable of understanding user requests in natural language and autonomously interacting with the uis of windows-based applications. the potential of such an agent is compelling, as it promises to simplify both routine and creative tasks, significantly reducing the user's effort for learning and operating within various applications. however, windows presents unique challenges due to its diverse applications, larger screen resolutions, and more complex guis compared to platforms like mobile or linux. a naive agent may struggle to understand and interact with the complex interfaces, making it difficult to fulfill user requests, especially those that span multiple applications. to address this gap, we introduce ufo , a specialized ui-focused vlm agent designed for seamless interaction with the windows os. we illustrate this concept at a high level in figure 1 , where ufo composes and sends an email by integrating text extracted from a word document, observations from an image, and summaries from a powerpoint presentation. ufo employs a dualagent framework that utilizes a divide-and-conquer approach to fulfill user requests. in this framework, the hostagent decomposes the user request into sub-tasks, while the appagent executes the assigned sub-tasks within individual applications. each agent utilizes screenshots and extracts information from the gui to make informed decisions. to ground actions in the application, ufo incorporates a control automator component specifically tailored for the windows os. this inspects control elements and translates actions from the agents into grounded executions within applications. this setup ensures reliable action execution and complete automation without the need for human intervention. as user requests often involve interactions across multiple applications, ufo includes a switching mechanism within the hostagent to facilitate transitions between different applications. this capability allows ufo to handle complex, multiapplication tasks that are frequently overlooked by other agents. additionally, ufo is highly extensible, allowing users to design and customize actions for specific tasks and applications. it also incorporates a safeguard feature to prevent the execution of risky actions. these enable ufo to automate lengthy and tedious processes through natural language commands while ensuring safety. we conducted comprehensive testing of the ufo framework on our windowsbench benchmark which include 50 tasks involving 9 widely used windows applications to capture a range of scenarios reflective of users' daily needs. the evaluation included both quantitative metrics and detailed case studies, demonstrating that ufo signif-icantly outperforms other gui agents. this performance underscores the robustness and adaptability of our design, particularly in handling complex requests that span multiple applications. to the best of our knowledge, ufo represents the pioneering agent specifically tailored for general applications within the windows os. ## related work the use of multimodal large language models (llms) for navigating and controlling guis in applications has become a prominent area of research. appagent @xcite leverages gpt-4v to mimic smartphone users, enabling actions on mobile applications based on smartphone snapshots, thereby autonomously fulfilling user requests on physical devices. mobileagent @xcite enhances the capabilities of gpt-4v within a similar mobile agent framework by integrating optical character recognition (ocr) tools, allowing it to achieve task completion rates comparable to human performance. in contrast, os-copilot extends this concept to desktop environments, using selfdirected learning to continuously evolve the agent's capabilities. cradle applies dino @xcite and sam @xcite for control detection within applications, further extending these techniques into the realm of video game playing @xcite . distinguishing itself from existing frameworks, ufo stands out through its unique features: (i) it is tailored to the windows os with specialized control inspection and interaction; (ii) it employs a dual-agent, divide-and-conquer approach for handling cross-application tasks; and (iii) it incorporates several design elements to ensure both customization and safety. ## the design of ufo we present ufo , a ui-focused agent designed for the windows os interaction. ufo comprehends users' requests expressed in natural language, and observe the ui screenshots of applications and operates on their control elements to fulfill the overall objective, transcending the boundaries of different applications. ## conclusion we present ufo , a ui-focused agent designed to fulfill user requests in natural language by intelligently interacting with applications on the windows os. ufo analyzes gui and control information to select the most appropriate application and element to execute actions that satisfy user queries. with its dual-agent architecture, ufo efficiently manages complex task switching between applications, enabling the completion of complex tasks that span multiple software environments. key features such as action customization and safeguard further enhance ufo's extendibility and safety. evaluation results from 50 requests across 9 popular windows os applications demonstrate its exceptional versatility and generalizability. to the best of our knowledge, ufo is the first ui automation agent specifically tailored for the windows os.