Hidden Forms: A Dataset to Fill Masked Interfaces from Language Commands (2025)

Abstract: This paper introduces Hidden Forms (hFORMS), a dataset of natural language commands paired with user interfaces with masked visual context. By obscuring specific UI elements, the dataset challenges Computer-Using Agents to parse natural language instructions and infer the correct bounding box locations by leveraging UI context. Furthermore, hFORMS contains three distinct masking strategies representing progressive difficulty levels. Additionally, we explore parameter-efficient fine-tuning approaches using Vision-Language models from the Llama and Qwen series, demonstrating that fine-tuning on mobile domains results in more than 5x improvement in zero-shot domain adaptation performance when identifying bounding boxes on the desktop and web domains.

## introduction recent work in nlp has seen the extension of language modeling techniques to develop computer-using agents (cuas) @xcite @xcite . cuas execute natural-language user requests by interacting with elements of the graphical user interface (gui), such as buttons, menus, and text fields. current cuas perform about half as well as humans on popular agent evaluation benchmarks such as os-world @xcite , webarena @xcite , and visualwebarena @xcite with the average human performance hovering around 75%. furthermore, leaderboards for these benchmarks 1 are populated by proprietary models, or models that use a significant number of parameters (> 70b) @xcite , raising concerns about locally deployable solutions. this requires developing parameter-efficient techniques with smaller language models. figure 1 : an example of the task in hforms. the task is to identify the bounding box location for the content on the screen. the interface contains multiple elements that are hidden from the model. motivated by the prevalence of mobile interactions, several datasets have been developed to build cuas in the mobile domain @xcite @xcite . however, leaderboards for these datasets are also dominated by closed-source systems such as gpt-4, claude, and gemini. locally deployable systems require developing smaller-scale models that run on lowresource hardware, e.g. single-gpu devices. however, building generalist ui-understanding capabilities in smaller vision-language models requires additional domain-specific training. to address these limitations, we build upon existing paired natural-language gui datasets and introduce hidden forms foot_0 (hforms). hforms addresses a crucial auxiliary task in ui understanding: given a natural language description of gui content or an action command, the system must infer the correct bounding box location to place this information. importantly, the ground truth location of the element is concealed from the system, compelling it to leverage on-screen contextual cues to successfully complete the task. ## related work prior work on ui modeling focused on the identification and classification of visual elements on mobile screens @xcite @xcite . following the development of vision-language models, more recent work focused on jointly modeling referring expressions within the context of mobile interfaces @xcite @xcite . uibert @xcite consists of a dataset specifically for the task of ui understanding. @xcite introduce five tasks to learn representations by jointly modeling on-screen content and their captions obtained through ocr. they evaluate trained models for referring expression retrieval, a multiple-choice task where the goal is to retrieve the correct on-screen content given a natural language description. screenqa @xcite is a dataset of questions and answers targeting content across multiple android apps. given an app screenshot, crowdworkers write questions and answers that address specific screen components. screenspot @xcite is a benchmark dataset of screenshots and instructions from ios, android, windows, macos, and webpages that evaluates the ability of models to identify the locations of screen content corresponding to natural language commands. as the names indicate, the splits are built by modifying screenqa, uibert, and screenspot. while screenqa and uibert contain android app screenshots from rico @xcite , screenspot is a smaller dataset consisting of screenshots from webarena @xcite . ## conclusion this work introduces hidden forms (hforms), a dataset comprising natural language commands paired with user interfaces where relevant information is masked to help build ui understanding capabilities in computer using agents. by obscuring ui elements, we challenge agents to parse natural language instructions and infer the correct bounding box locations by leveraging contextual cues. hforms presents three distinct masking strategies representing progressive difficulty levels. additionally, we explore parameter-efficient fine-tuning approaches using vision-language models from the llama and qwen series, demonstrating that finetuning on mobile domains significantly improves zero-shot domain adaptation performance on the desktop and mobile domains.