Understanding GUI Agent Localization Biases through Logit Sharpness (2025)

Abstract: Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions. Despite their promising performance, these models frequently exhibit hallucinations—systematic localization errors that compromise reliability. We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics. To better quantify model uncertainty, we introduce the Peak Sharpness Score (PSS), a metric that evaluates the alignment between semantic continuity and logits distribution in coordinate prediction. Building on this insight, we further propose Context-Aware Cropping, a training-free technique that improves model performance by adaptively refining input context. Extensive experiments demonstrate that our framework and methods provide actionable insights and enhance the interpretability and robustness of GUI agent behavior.

## introduction recent progress in large language models (llms; @xcite @xcite @xcite @xcite has greatly advanced natural language understanding and generation. however, their reliance on purely textual inputs and outputs limits their applicability in perceptual and interactive tasks. to address this, multi-modal large language models (mllms) have emerged by incorporating visual inputs alongside text. models such as flamingo @xcite , gemini @xcite , and qwen-vl @xcite @xcite enable more comprehensive reasoning across modalities, supporting tasks like visual question * corresponding author: jing tang. building on recent advances in llms and mllms, researchers have turned to gui agents-intelligent systems capable of autonomously operating graphical user interfaces via perception, reasoning, and action. these agents must interpret screen layouts, understand task instructions (often in natural language), and generate accurate sequences of interface actions such as clicks or keystrokes. effective gui agents require the integration of language understanding, visual perception, and action planning in dynamic environments. a key challenge lies in accurately identifying interaction targets on the interface. general-purpose mllms often fall short in gui-specific tasks, particularly in predicting precise operation coordinates. to overcome this, recent approaches adopt training-based pipelines that enhance the agent's capabilities through continued pre-training on large auxiliary datasets, followed by the integration or adaptation of neural modules tailored for gui tasks. this foundation enables more effective fine-tuning on smaller, domain-specific datasets, improving precision and robustness in real-world gui interactions. existing gui agent models are typically pretrained or fine-tuned on large-scale datasets of interface operations. however, they often underperform on tasks that are trivial for human users. moreover, these models exhibit weak localization accuracy for rarely seen or uncommon icons, which hinders their ability to generalize across diverse interface environments. in this study, we systematically analyze the forms of hallucination exhibited by existing gui agent models. these hallucinations often manifest as inaccurate or implausible predictions during icon localization, especially in cases involving unfamil-iar interface elements. to facilitate a controlled investigation, we introduce a novel icon library containing symbols that are semantically clear and visually distinctive, yet rarely encountered in common human-computer interaction scenarios. these icons are designed to integrate naturally into gui layouts while providing new challenges for model generalization. to better characterize and quantify hallucinations in gui localization tasks, we propose formal definitions and a taxonomy of hallucination types, along with corresponding classification algorithms. this framework allows for a more precise evaluation of model behavior when interacting with both familiar and unfamiliar interface elements. in parallel, we examine the distribution of logit scores produced by gui agent models during coordinate prediction. unlike conventional natural language tasks such as question answering, gui localization tasks typically involve output tokens that represent numeric values (e.g., x and y coordinates). these tokens exist in an ordered space, where semantic proximity is inherently meaningful. for instance, when predicting the token "6", surrounding tokens like "5" and "7" are expected to have higher logits due to their closeness in both numerical and spatial terms, whereas tokens such as "1" or "9" are more distant in this context. this characteristic provides a unique opportunity to study structured output spaces and the nature of model uncertainty in gui interaction tasks. in addition, informed by our analysis, we introduce an image cropping strategy that effectively reduces hallucinations and enhances the coordinate prediction performance of existing gui agent models. this work makes several key contributions: gui agents gui agents @xcite @xcite @xcite , are designed to interact with graphical user interfaces through visual perception, natural language understanding, and action planning. early generalist agents such as gato @xcite demonstrated multitask capabilities across robotic control, games, and web interfaces, but lacked fine-grained spatial grounding. recent approaches have expanded gui agent capabilities through stronger visual grounding and multimodal reasoning. webgum @xcite introduces a hierarchical planning framework that combines llms with execution modules and perceptual grounding. omniparser @xcite uses auxiliary visual models to mark the position of elements in the operation interface, improving the performance of gpt-4v in gui agent tasks. showui @xcite and uground @xcite synthesize a large amount of training data for training efficient gui agent models. despite progress, many agents still struggle with localization of unfamiliar ui elements and suffer from hallucination-like errors, especially in low-resource or distribution-shifted settings. this motivates continued research into robust visuallanguage grounding, more diverse pretraining data, and better uncertainty modeling in gui environments. ## investigating hallucinations in gui agent models most existing benchmarks emphasize task success and interaction accuracy with gui elements, but often overlook a nuanced analysis of hallucinations-systematic or implausible errors arising during model interaction. these evaluations typically rely on screenshots and provide only the ground truth bounding box, neglecting the potential influence of surrounding interface elements. to address this gap, we design a series of controlled experiments aimed at systematically analyzing the hallucination behavior of gui agents. in addition, we introduce a dedicated classification algorithm that categorizes hallucinated outputs, providing a finer-grained understanding of error types and their underlying causes in gui-based localization and interaction tasks. specifically, we classify model responses into four distinct categories: correct response, biased hallucination, misleading hallucination, confusion hallucination. ## new metric: peak sharpness score (pss) while the previous section examined hallucination classification and spatial errors in gui agent coordinate predictions, the confidence levels of these models in producing such outputs remain underexplored. conventional evaluation metrics, such as accuracy and token-level perplexity, fail to adequately capture this aspect, particularly in multimodal tasks requiring spatial reasoning and action grounding. to address this, the subsequent section introduces a novel confidence-oriented metric designed specifically for gui agent models. this metric provides a more nuanced and detailed evaluation of model certainty during task execution, enabling improved diagnosis of failure modes and informing future enhancements. traditional qa task where is the capital of france? paris no semantic continuity gui agent task close the window [0.52, 0.56] key token ## conclusion we present a controlled experimental framework for evaluating gui agent models via an icon localization task with fully annotated interface elements. this setup enables fine-grained classification of model responses, revealing diverse hallucination patterns beyond binary accuracy. to better characterize model uncertainty, we introduce the peak sharpness score (pss), a metric that quantifies the alignment between semantic continuity of coordinate tokens and the smoothness of logit distributions. empirical results show that different hallucination types correspond to distinct pss profiles, offering insights into their underlying causes. additionally, we propose context-aware cropping, a simple yet effective method that improves localization performance and model confidence without retraining. our findings highlight the importance of structured evaluation and representation-aware metrics in advancing robust gui agents.