Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM -Powered Mobile GUI Agents (2025)

Abstract: Graphical user interface (GUI) agents powered by multimodal large language models (MLLMs) have shown greater promise for human-interaction. However, due to the high fine-tuning cost, users often rely on open-source GUI agents or APIs offered by AI providers, which introduces a critical but underexplored supply chain threat: backdoor attacks. In this work, we first unveil that MLLM-powered GUI agents naturally expose multiple interaction-level triggers, such as historical steps, environment states, and task progress. Based on this observation, we introduce AgentGhost, an effective and stealthy framework for red-teaming backdoor attacks. Specifically, we first construct composite triggers by combining goal and interaction levels, allowing GUI agents to unintentionally activate backdoors while ensuring task utility. Then, we formulate backdoor injection as a Min-Max optimization problem that uses supervised contrastive learning to maximize the feature difference across sample classes at the representation space, improving flexibility of the backdoor. Meanwhile, it adopts supervised fine-tuning to minimize the discrepancy between backdoor and clean behavior, enhancing effectiveness and utility. Extensive results show that AgentGhost is effective and generic, with attack accuracy that reaches 99.7% on three attack objectives, and shows stealthiness with only 1% utility degradation. Furthermore, we tailor a defense method against AgentGhost that reduces the attack accuracy to 22.1%.

## introduction graphical user interface (gui) agents powered by multimodal large language models (mllms) have demonstrated significant potential in real-world interactions @xcite . by incorporating various capabilities, such as environment perception @xcite @xcite , planning @xcite , memory @xcite , and reflection @xcite @xcite , the effectiveness of task completion is significantly enhanced. meanwhile, gui agents also focus on encapsulating atomic operations into application programming interfaces (apis) to improve task execution efficiency @xcite @xcite . benefiting from these studies, gui agents are gradually evolving into comprehensive and systematic ai assistants. however, gui agents have revealed numerous security vulnerabilities due to possessing elevated privileges @xcite @xcite . @xcite and @xcite found that gui agents can be distracted by environmental context (e.g., pop-ups). further studies on adversarial attacks show that carefully crafted inputs or environmental injections can hijack gui agents @xcite @xcite , causing unexpected behaviors @xcite . however, the success rates of adversarial attacks are relatively low, as task-tuned mllms tend to follow user goals, making hijacking difficult. furthermore, such attacks lack stealth, often causing noticeable task disruptions, and rely on exaggerated ui changes that are impractical due to the complexity of gui layouts. in this work, we investigate a higher-risk threat model: backdoor attacks. due to the high tuning cost of mllm-based gui agents (e.g., os-atlas requires 32 × 80 gb and 17 hours of training), users often rely on open-source models or api from third-party platform. the widespread adoption of such models, evidenced by downloads attack behaviors triggered by a combination of user goals and interaction episodes (related to history actions). the remaining two attack behaviors, concerning environment status and task progress, are illustrated in figure 5 . therefore, we propose agentghost, a stealthy, flexible, and effective backdoor attack, as illustrated in figure 1(b-c ). specifically, gui agents can naturally expose multiple backdoor injection conditions to the attacker in the episode, such as history steps, environment status, and task progress. based on this observation, we define a combination trigger to improve stealthiness-activating agent-ghost only when both goal-and interaction-level conditions are met; otherwise, it behaves normally. for flexibility, we introduce multiple combination triggers to form backdoor mappings with various attack targets, such as system paralysis, malicious network induction, and privacy leakage, quantifying the concurrent impact of agentghost. agent-ghost poses significant threats to practical gui agents. for example, when the user sends an alarm clock to set a goal, agentghost performs an additional step of attack action like a hidden ghost hand, but the user goal is also completed successfully. furthermore, we define a min-max optimization to satisfy the attack goals. agentghost exploits supervised contrastive learning (scl) @xcite to orthogonalize poisoned samples from di-2 accessed on 2025.9.5 from https://huggingface.co/ os-copilot/os-atlas-base-7b ## related work in this section, we review related work that underpins this study, focusing on gui agents, their security vulnerabilities, and backdoor defenses. ## pilot study in this section, we formalize the problem statement. we then analyze gui agents' backdoor vulnerabilities in the context of existing attack methods. ## methodology this section introduces agentghost. we first describe its threat model in section 4.1, followed by a detailed explanation of the method in section 4.2. ## experiments this section will introduce the experimental setup and show our empirical results with findings, including the effectiveness and utility of agentghost. ## conclusion in this work, we introduce agentghost, a stealthy and effective backdoor attack framework that re-veals a critical vulnerability in mllm-powered gui agents. agentghost defines three attack objectives and adopts a min-max optimization strategy to inject composite backdoors at both the goal and interaction levels. extensive experiment results show that agentghost achieves up to 99.7% attack success across all objectives with only 1% utility degradation, and generalizes across various models and mobile gui benchmarks. to alleviate agent-ghost, we also introduce a self-reflection defense that reduces amr to 22.1%, offering valuable insights for strengthening the security of mllmbased gui agents.