Visual Grounding for User Interfaces (2024)

Abstract: Enabling autonomous language agents to drive application user interfaces (UIs) as humans do can significantly expand the capability of today’s API-based agents. Essential to this vision is the ability of agents to ground natural language commands to on-screen UI elements. Prior UI grounding approaches work by relaying on developer-provided UI metadata (UI trees, such as web DOM, and accessibility labels) to detect on-screen elements. However, such metadata is often unavailable or incomplete. Object detection techniques applied to UI screens remove this dependency, by inferring location and types of UI elements directly from the UI’s visual appearance. The extracted semantics, however, are too limited to directly enable grounding. We overcome the limitations of both approaches by introducing the task of visual UI grounding, which unifies detection and grounding. A model takes as input a UI screenshot and a free-form language expression, and must identify the referenced UI element. We propose a solution to this problem, LVG, which learns UI element detection and grounding using a new technique called layout-guided contrastive learning, where the semantics of individual UI objects are learned also from their visual organization. Due to the scarcity of UI datasets, LVG integrates synthetic data in its training using multi-context learning. LVG outperforms baselines pre-trained on much larger datasets by over 4.9 points in top-1 accuracy, thus demonstrating its effectiveness.

## introduction autonomous language agents that are capable of interacting with real-world applications are emerging @xcite @xcite @xcite . provided with a task described in natural language, these agents drive application user interfaces as humans do by clicking, typing, scrolling, etc. the * work done while at microsoft research. prior work @xcite ) achieves ui grounding by assuming the location bounds and types of ui elements present in a screen are known beforehand. hence, they define grounding as the problem of ranking a set of ui elements based on the given natural language command. the set of ui elements is computed automatically using developer-provided ui metadata, consisting of ui trees (e.g., web dom tree or android view hierarchy) and accessibility annotations. the issue with this approach is that such ui metadata is often not accessible for security or privacy reasons @xcite . developer-provided metadata can also be noisy, corrupted with missing object descriptions or misaligned structure information @xcite . finally, as others pointed out @xcite , accessibility labels are generally not provided for all ui elements (see appendix a.1 for further details). these constraints make these approaches hard to deploy and limit their performance. another way of approaching this problem without relying on ui metadata is to train object detection models for ui screens @xcite . this line of work, generally referred to as screen understanding or screen parsing, localizes ui elements in a screen solely from its visual appearance. elements are labeled with technical terms such as "button", "text-input", "icon", etc. as these labels carry limited semantic information, they are not sufficient to directly support grounding of natural language commands. this means that a second model, possibly an llm, must figure 1 : visual ui grounding unifies the task of ui grounding that relies on the availability of ui metadata and screen understanding which localizes elements in a screen and classifies them into pre-defined types (button, text-label, text-input, icon, etc.). in this new task, a ui element referenced by a natural language command must be localized in the given ui screen, by relying solely on the screen's visual appearance (without assuming ui metadata). we address the limitations of both worlds by unifying detection and grounding into the new task of visual ui grounding, illustrated in fig. 1 . the model takes as input a ui screenshot (without metadata) and a free-form language expression, and must predict the bounding box of the referenced element. hence, unlike previously-proposed methods where bounding boxes of candidate ui objects are given as input or pre-computed by a separate model, here a single model must perform both element detection and action grounding. in the search for a solution to the problem of visual ui grounding, we first consider recent work on open-vocabulary object detection @xcite @xcite . these models are trained end-to-end to map natural language expressions to objects in an image. while they represent a perfect fit for our problem, we find that, despite their large training datasets, they do not perform well on ui screens (see baseline comparisons in §5). our explanation is that these models are trained on real-world scene datasets @xcite where objects of the same appearance (color, shape, size) share sim-ilar meanings, whereas ui objects are subject to application and context sensitivity. in other words, in ui screens, objects that may look similar have different meanings depending on the application and surrounding ui elements. for example, consider a heart icon which in facebook loves a post, but in etsy adds a product to the favorites; if the same icon appears next to a label "click for more" it assumes yet another meaning. to address the problem of application and context-sensitivity of ui objects, we propose lvg (layout-guided visual grounding). we observe that while objects in real-world scenes do not usually follow a regular pattern in their arrangement, ui elements are organized through layouts, which can be key to understanding their meaning. for example the function of an icon or an element in a grid can be better understood by relating it to a nearby text label or to another element spatially aligned to it. hence, we introduce layout-guided contrastive learning where the model learns to classify elements into groups based on their visual containers (headers, lists, tables, etc.). this enforces the target element's features to be closer to those of its sibling elements and far from those of elements in other containers, thus enriching their semantic representations. application-derived features are then combined with element-specific features. further, to cope with the lack of ui grounding datasets, we synthetically generate natural language referring expressions paired with original ui screens. we successfully transfer knowledge learned from synthetic to real-user expressions using multi-context learning, i.e., forcing the model to generate similar features when synthetic and natural expressions are referring to the same element. in summary, we make the following contributions: (i) we define the task of visual ui grounding, (ii) we propose a solution, lvg, and introduce layout-based contrastive learning, and (iii) we generate a synthetic dataset of diversified language queries and use it effectively through multi-context learning. overall, lvg surpasses strong baselines by over 4.9 points on top-1 accuracy. screen understanding screen understanding (also called screen parsing) models avoid the dependency on ui metadata, by inferring bounding boxes and types of on-screen elements solely from a ui screenshot @xcite @xcite . the inferred class labels ("button", "radio-button", "slider", "text-input", etc.), however, are semantically very limited to directly enable grounding of open-vocabulary referring expressions. for this reason these methods must be paired with a second model, an llm or vlm, for language grounding. @xcite use screen understanding techniques based on a combination of ocr and iconnet @xcite to detect elements on the screen and produce a textual representation of the ui. then, they train a grounding model using behavioural cloning or use llms in a zero/few-shot manner to identify the referenced element. another 2-step approach @xcite which involves gpt-4v uses the same screen understanding techniques to identify bounding boxes of relevant elements, which are then represented by visually adding numeric tags to the ui image @xcite . finally, pix2act @xcite adopts pix2struct @xcite (consisting of an image encoder and text decoder) to first transform ui screenshots of miniwob @xcite synthetic webpages into simplified html and then apply behavioural cloning, reinforcement learning or monte carlo tree search. the main downside of these approaches is that the preliminary step of converting ui screenshots into textual representations or annotating ui images with numeric tags causes information loss. some elements may be missed, and especially text-only representations are not well suited for visual elements such as icons and symbols. the two-step approach also increases the deployment costs from one model to two. in our approach, one model is trained end to end, thus lowering the deployment costs and avoiding any lossy pre-processing. open-vocabulary object detection recent work in the computer vision community tackles the problem of open-vocabulary object detection @xcite @xcite @xcite , where a model is tasked to detect classes of objects that have not been introduced to it before. regionclip @xcite ) learns a regional visual-semantic space that covers rich object concepts such that it can be used for open-vocabulary object detection. glip @xcite unifies grounding and detection tasks by reformulating object detection as phrase grounding, thus being able to learn from both detection and grounding datasets. while related to our goal, these methods are designed for images and objects that represent real-world scenes. when fed with ui datasets their performance is inferior because ui screens exhibit some unique features (see results in §5 and appendix a.3). to address ui-specific challenges we introduce layoutguided contrastive learning and leverage globallocal feature aggregation. ## method a key contribution of our work is to address the problem of application and context sensitivity which characterizes ui screens. application sensitivity occurs with ui elements that despite their similar appearance have different functionality in different applications (e.g., a "hand" symbol in a video call application or in a drawing application have completely different functions). context sensitivity occurs with ui elements that change their functionality depending on "context", i.e., neighboring ui elements (e.g., a list item must be considered in the context of the other items appearing in the same list or a text-label can change the meaning of a symbol located next to it). next, we describe how lvg addresses these challenges. fig. 2 shows the architecture of lvg. we use swin transformer @xcite as the visual backbone n v , to extract visual features f v from ui screens, and bert @xcite as text backbone n t , to extract textual features f t from natural language commands. ## datasets and data synthesis for training, we use the uibert dataset @xcite , foot_0 which contains 16,660 referring expressions associated with a total of 5,682 android ui screenshots. we also complement this humancollected dataset with a synthetic dataset. we obtain ui screens of original android apps from the rico dataset @xcite and use the ui tree information associated with each screenshot to determine a set of cues from which we heuristically generate referring expressions. our cues extend those proposed in ricosca @xcite where every expression consists of an operation (a verb, such as "tap") and a target element. we make various improvements to ricosca to increase the diversity of the generated expressions, and add layout-based cues. we generate expressions only for interactable ui elements (buttons, input fields, icons, etc.) through the following process. first, we assemble a collection of operational phrases such as "click xxx", "select xxx", "type xxx", "tap xxx", "go to xxx". each phrase consists of a verb and a placeholder xxx. second, we establish a set of rules to replace "xxx" placeholders with one or multiple object identifying expressions. these expressions are generated using ui tree information. for example, a ui tree may list an object of type "button", with name "cancel", with location bounds x 1 ,y 1 ,x 2 ,y 2 , and with property @xmath0. we create rules to produce object expressions such as "the button with name cancel" or "the cancel button" or simply "cancel". in general, we identify a target element using its name (accessibility label, textual content), type (class name) or location. we generate location-based object expressions using the location bounds of the object and the neighboring objects to obtain object descriptions such as "at the top of the page" (using absolute location) or "appearing in the menu next to the login button" (using relative location). third, we create multiple rules based on the object's properties to determine which operational phrases can be applied to an object. for example, if the object's property "clickable" is set to true and its type is "button", operational phrases such as "click xxx" or "tap xxx" can be applied to it. finally, for each object we assemble multiple referring expressions. the selected operational phrases are instantiated using one or multiple object expressions. for instance, the phrase "tap xxx" selected for the "cancel" element described above is instantiated as "tap cancel" or "tap the cancel button" or "tap the cancel button next to login". fig. 4 shows some examples of generated synthetic expressions. overall, we generated 22,617 synthetic expressions for 21,282 android ui screens. we found that simply mixing uibert's real-user expressions with synthetic ones did not bring noticeable improvements due to the domain gap (synthetic expressions can be longer and the ratio of referring queries using relative location is higher). we adopt multicontext learning which in robotics has been shown to successfully combine together imitation learning datasets of different sizes and nature @xcite . we find it is important to generate for each uibert expression a synthetic counterpart, for the same referred element. this forces the model to map both types of expressions to the same space, and to ultimately leverage the larger size of synthetic data. ## evaluation we train and evaluate on the uibert dataset @xcite using the official splits. we expand the uibert train set with 22,617 synthetic expressions. as evaluation metric we use acc@k with @xmath1.5, which measures the fraction of correctly identified ui elements in the top k ranked results. we compare against 3 baselines: glip @xcite , ofa @xcite , and uninext @xcite . (uninext and ofa currently rank first and third, respectively, in the refcoco leaderboard @xcite .) all models are trained on uibert with or without synthetic data ## conclusions we propose the new task of visual ui grounding and present our solution to it. compared to strong baselines trained on much larger datasets, lvg's layout-guided contrastive learning and multi-context approach for synthetic data demonstrate great improvements in identifying ui elements referenced by nl expressions.