OS -Kairos: Adaptive Interaction for MLLM -Powered GUI Agents (2025)

Abstract: Autonomous graphical user interface (GUI) agents powered by multimodal large language models have shown great promise. However, a critical yet underexplored issue persists: over-execution , where the agent executes tasks in a fully autonomous way, without adequate assessment of its action confidence to compromise an adaptive human-agent collaboration. This poses substantial risks in complex scenarios, such as those involving ambiguous user instructions, unexpected interruptions, and environmental hijacks. To address the issue, we introduce OS-Kairos , an adaptive GUI agent capable of predicting confidence levels at each interaction step and efficiently deciding whether to act autonomously or seek human intervention. OS-Kairos is developed through two key mechanisms: (i) collaborative probing that annotates confidence scores at each interaction step; (ii) confidence-driven interaction that leverages these confidence scores to elicit the ability of adaptive interaction. Experimental results show that OS-Kairos substantially outperforms existing models on our curated dataset featuring complex scenarios, as well as on established benchmarks such as AITZ and Meta-GUI, with 24.59%~87.29% improvements in task success rate. OS-Kairos facilitates an adaptive human-agent collaboration, prioritizing effectiveness, generality, scalability, and efficiency for real-world GUI interaction. The dataset and codes are available at Anonymous.

## introduction multimodal large language models (mllms) have been explored to develop graphical user interface (gui) agents capable of analyzing the screen and performing human-like behaviors on operating systems @xcite @xcite . existing efforts in building gui agents have focused on the autonomous mode @xcite @xcite , with improved capabilities such as grounding @xcite and reasoning @xcite @xcite . despite exciting progress, we observe that existing gui agents exhibit significant over-execution issues -the agent executes tasks in a fully autonomous way, without adequate assessment of its action confidence to compromise an adaptive human-agent collaboration. as shown in figure 1 (a), popular gui agents such as os-atlas @xcite are unable to click the filters button correctly, causing unexpected interruptions and task failure. over-execution poses significant challenges in complex real-world scenarios (examples shown in figure 2 ), highlighting fundamental limitations in current gui agents. first, ambiguous instructions from the user leads to information absence in gui automation (e.g., account logout). second, existing gui agents depend heavily on the foundation mllms and therefore suffer from unexpected interruptions when executing complex instructions. besides, these models will also generate hallucinations @xcite and shortcut predictions @xcite . third, environmental hijacks, such as network connection failure and pop-up hijacking) @xcite . to address these challenges, we are motivated to integrate confidence scoring into the foundation model, allowing adaptive human intervention for gui agents (figure 1(b) ). concretely, we introduce os-kairos, an adaptive gui agent capable of predicting confidence levels at each interaction step and efficiently determining whether to act autonomously or seek human intervention. os-kairos incorporates two key mechanisms: (i) collaborative probing that annotates confidence scores at each interaction step; (ii) confidence-driven interaction that utilizes these confidence scores to enhance the ability of adaptive interaction. to annotate the confidence scores for the probed gui agents in real-world scenarios, we first design a collaborative confidence probing framework. inspired by @xcite , this framework integrates a layout parsing model @xcite and the most capable proprietary model, gpt-4o @xcite to function as a critic model. the critic model is used to supervise plan scheduling and confidence score based on our curated instructions to address complex scenarios. this framework is the first toolkit designed to identify when human intervention is necessary, generate confidence scores, and facilitate the automated construction of gui trajectories. to further integrate confidence scoring into the probed gui agent, we validate and refine these gui trajectories and then fine-tune the model. this approach ensures action prediction accuracy while improving adaptability of human intervention. experimental results in complex scenarios show that os-kairos achieves state-of-the-art performance with action type success rate of 99.88%, action success rate of 95.90%, and task success rate of 88.20%. also, we confirm os-kairos's effectiveness on two well-established gui benchmarks: meta-gui @xcite and aitz @xcite . comprehensive analysis reveals that os-kairos prioritizes effectiveness, generality, scalability, and efficiency, making it a competitive agent for real-world gui interactions. our work makes the following key contributions: ## related works our work falls into the field of mllm-powered agents. this section will first review the recent progress in building gui agents and then discuss the capability probing approaches for gui agents. ## pilot experiments in this section, we first define gui agent paradigms and then investigate the over-execution issue of the existing gui agent in three complex scenarios. as shown in figure 2 , gui agents confront substantial risks in real-world scenarios, such as those involving ambiguous user instructions (e.g., information absence and account logout), unexpected interruptions (e.g., hallucinations and shortcuts), and environmental hijacks (e.g., pop-up hijacking and permission unauthorized). ## methodology this section presents os-kairos. we first introduce a collaborative probing framework that dynamically annotates the confidence scores at each interaction step. then, we will describe confidencedriven interaction that integrates confidence scoring into gui agents, resulting in adaptive interaction. figure 3 shows an overall illustration. ## experiments this section will introduce the experimental setup, followed by our empirical results and analysis.