L exi: Self-Supervised Learning of the UI Language (2022)

Abstract: Humans can learn to operate the user interface (UI) of an application by reading an instruction manual or how-to guide. Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text. We explore how to leverage this data to learn generic visio-linguistic representations of UI screens and their components. These representations are useful in many real applications, such as accessibility, voice navigation, and task automation. Prior UI representation models rely on UI metadata (UI trees and accessibility labels), which is often missing, incompletely defined, or not accessible. We avoid such a dependency, and propose Lexi, a pre-trained vision and language model designed to handle the unique features of UI screens, including their text richness and context sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k UI images paired with descriptions of their functionality. We evaluate Lexi on four tasks: UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition.

## introduction over the years, humans have learned to parse application user interfaces (uis). with only a glance at a previously unseen ui they can understand its elements and identify those relevant to a desired task. to some extent, humans have learned the "ui language". the words of this language are ui elements, which can be visual (an icon), textual (a hyperlink), or both (a text button). if machines could parse and understand the ui language in the same way, they could map natural language commands to ui elements, facilitate access to uis for visually-impaired users, and ultimately operate uis on behalf of users. ## uicaption dataset we curate a new dataset of ui images (screenshots and icons) paired with functional captions. we collect ui images, alt-text, and instructions accompanying a ui image from three sources: (i) instruction manuals extracted from support websites of 23 microsoft (ms) apps (teams, outlook, excel, word, etc.) and zoom. (ii) android support websites, whose urls are obtained from the an-droidhowto dataset @xcite , and (iii) 58k general tech websites whose urls are obtained by submitting search queries containing technical commands (extracted from ms and android manuals) to google image search and inspecting the top 60 results. to improve data quality, we perform data filtering, including a simple hash-based and text-based near-duplicate image removal, blurred image removal, and image relevance (e.g., excluding content-unrelated images such as facebook and twitter icons). for each collected image, we use the text surrounding it in the webpage to generate from 1 up to 5 functional captions (the number depends on the length and structure of the surrounding text). if available, we also append the alt-text. rather than describing the appearance of an image, functional captions describe the functionality of one or multiple ui elements, in the context of other elements. textual context may translate into spatial context because the instructions to accomplish a task often involve co-located ui elements. for example, in figure 1b , the "video call" and "audio call" icons are associated with the caption "select video call or audio call to start a call"; the instruction defines a logical relationship between the two functions, which corresponds to a spatial relationship, i.e., the two icons are next to each other in the ui. we thus obtain the uicaption dataset consisting of 113,971 unique ui images and 133,817 image-caption pairs. this dataset significantly differs from those used in prior ui understanding models @xcite . the data is extremely varied in terms of applications (excel, gmail, chrome, photos, etc.), platforms (mobile & desktop), number of ui elements (from single icons to complete desktops), text density (word documents, photos, or excel spreadsheets), pixel density, and quality (e.g., screenshots may not be pristine or may have been annotated, such as in figure 1a ). more details on the dataset and the crawling process can be found in appendix a. o . 0/1 co-attentional transformer layers … place a call [alt] ## method lexi follows the two-stream transformer-based architecture of various vl models @xcite . as illustrated in figure 2 , it consists of an image encoder v , a text encoder l, and co-attentional transformer layers for cross-modal representation learning. given a text input t represented as a set of word tokens w 1 , ..., w t and an image i represented as a set of region features v 1 , ..., v m augmented with their corresponding detected texts, lexi outputs final representations h v0 , ..., h vm for the visual stream and h w0 , ..., h wt for the textual stream. the most distinct aspect of lexi is the early-infusion of detected text vector representations with corresponding visual features, which is motivated by the frequent presence of text in ui images. moreover, the vocabulary of functional captions including technical action verbs and specific named entities motivates an additional self-supervision task for pre-training. ## evaluation on downstream tasks we evaluate lexi through 4 downstream tasks. in this section, we describe our experimental setup, introduce datasets and baselines, describe each task's settings and results, and provide an ablation study. ## limitations in this section, we describe some limitations of our method, dataset, and data collection approach. our approach depends on pre-training using publicly available web data, which is crawled at scale and for a diverse set of applications. web data is highly skewed towards popular applications; less popular application might have less or lowerquality documentation and images online. in the future, we will focus on collecting ui data from other sources, such as instructional videos, online tutorials, and bug reports. we also observe that for certain application categories, such as shopping, transportation, weather, or news, instruction manuals are rare or totally missing. another limitation that arises from curating pretraining data from the web is how to ensure zero overlap between pre-training and downstream evaluation tasks. a url-based filter of the crawled websites does not eliminate overlaps as aggregator websites like wikihow and lifewire may contain manuals of different apps and platforms. partitioning data based on application names is also difficult as the name of the application is not explicitly encoded in the webpage source. our data and method have been evaluated for english instructions and uis that contain text in english. future work should expand to different languages. currently, the method is trained and evaluated on instructions whose length is at most 512 tokens. pre-training the model even on a modest dataset like uicaption already requires significant gpu resources. the obtained model is not optimized to run on edge devices like mobile phones, thus possibly raising privacy concerns. for some scenarios that motivate this work (e.g., accessibility support and voice-based navigation in mobile apps) it would be highly desirable to execute and update the models locally. ## conclusion we propose to learn generic representation models for the ui language from instruction manuals and how-to guides. we apply prior vl models to this data and encounter new challenges due to the text richness and context sensitivity of ui screens. to address them, we build a new dataset based on the concept of functional caption, design a new vision encoder which leverages text detected in uis, and create a new pos pre-training task. tested on four downstream tasks, lexi largely outperforms our baselines, including a multi-task vilbert model trained on much larger datasets. google is trying to limit what apps can use an accessibility service (again). https://www.xda-developers.com/google-tryinglimit-apps-accessibility-service/ . ning xie, farley lai, derek doran, and asim kadav. 2019. visual entailment: a novel task for fine-grained image understanding. arxiv, abs/1901.06706. semih yagcioglu, aykut erdem, erkut erdem, and nazli ikizler-cinbis. 2018. recipeqa: a challenge dataset for multimodal comprehension of cooking recipes. in proc. of the 2018 conference on empirical methods in natural language processing, pages 1358-1368. association for computational linguistics.