Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements (2020)

Abstract: Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general. Yet, these descriptions are constantly missing in mobile UIs. We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces. We collected a large-scale dataset for widget captioning with crowdsourcing. Our dataset contains 162,860 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens. We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions. The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces.

## introduction mobile apps come with a rich and diverse set of design styles, which are often more graphical and unconventional compared to traditional desktop applications. language descriptions of user interface (ui) elements-that we refer to as widget captionsare a precondition for many aspects of mobile ui usability. for example, accessibility services such as screen readers, e.g., @xcite , rely on widget captions to make ui elements accessible to visually impaired users via text-to-speech technologies. importantly, widget captions are an enabler for many language-based interaction capabilities such as voice commands and general screen understanding efforts. however, a significant portion of mobile apps today lack widget captions in their user interfaces, which have stood out as a primary issue for mobile accessibility @xcite . more than half of image-based elements have missing captions @xcite . beyond image-based ones, our analysis of a ui corpus here showed that a wide range of elements have missing captions. existing tools for examining and fixing missing captions @xcite @xcite @xcite require developers to manually compose a language description for each element, which imposes a substantial overhead on developers. we propose widget captioning, a novel task to automatically generate captions for ui elements foot_0 based on their visual appearance, structural properties and context (see figure 1 ). this task is analogous to image captioning that generates language descriptions for images, e.g., @xcite ; @xcite . however, widget captioning raises several unique challenges. user interfaces are highly structural while traditional image captioning mostly focus on raw image pixels. widget captioning is concerned with describing individual elements in the ui rather than the entire ui screen, while the entire screen provides useful contextual information for widget captioning. we target language generation for a broad set of ui elements, rather than only image-based ones. as we will show in our data analysis, many non-image elements also suffer the lack of captions. these challenges give rise to several interesting modeling questions such as how to combine both structural and image input and how to effectively represent each modality. we start by processing and analyzing a mobile ui corpus. we then create a large dataset for widget captioning by asking crowd workers to annotate a collection of ui elements in the corpus. based on this dataset, we train and evaluate a set of model configurations to investigate how each feature modality and the choice of learning strategies would impact caption generation quality. our champion model that is based on a transformer @xcite to encode structural information and a resnet @xcite for image input is able to produce accurate captions for ui elements based on both automatic and human evaluation. in summary, the paper makes the following contributions: ## related work the lack of captions or alternative text has been a universal problem in user interfaces, ranging from mobile apps to web pages @xcite @xcite . based on an analysis of an android ui corpus @xcite , @xcite revealed that a significant portion of image-based buttons lack alternative text. by examining a broader set of ui elements, we found missing captions is a general issue across ui element categories. automatic image captioning has been a classic task where a model learns to generate language descriptions for images, which has gained substantial progress with the advance of deep learning @xcite @xcite @xcite , and the availability of datasets such as flickr30k @xcite and ms-coco @xcite . in contrast to image captioning, widget captioning that we propose is concerned with describing individual elements in the context of the ui screen. in addition to image input, widget captioning has access to ui structures such as view hierarchies. these raise unique modeling opportunities for multimodal captioning. many image captioning models @xcite involve an encoding net and a language generation net. the encoding net is typically a deep convolutional neural network (cnn) @xcite that encodes the image input as a collection of latent vector representations. the generation net is often an autoregressive decoding model, enhanced with neural attention. for widget captioning, the encoding net needs to encode multimodal input that include both images and ui structures. for ui structure encoding, recent work @xcite investigated both graph convolutional network (gcn) @xcite and transformer @xcite and showed that a transformer encoder gives better performance on the task, which we will use in this work. our learning strategy is akin to bert @xcite that uses the words in the same sentence to predict those missing using transformer, to learn a contextual word representation. in our case, we use the information of elements in the same screen context to predict those with missing captions. to generate captions, based on the encoder output, we run multiple instances of the decoding model in parallel, one for each element to be captioned. ## creating the widget caption dataset we first create a mobile ui corpus, and then ask crowd workers to create captions for ui elements that have missing captions, which is followed by a thorough analysis of the dataset. ## widget captioning models to understand the challenges and feasibility of automatic widget captioning, we investigate deep models for this task. captioning models are often designed based on an encoder-decoder architecture. we formulate widget captioning as a multimodal captioning task where the encoder takes both the structural information and the pixel appearance of the ui element, and the decoder outputs the caption based on the encodings (see figure 7 ). ## experiments we first discuss the experimental setup, and then report the accuracy of our model as well as an analysis of the model behavior. ## conclusion we present widget captioning, a novel task for automatically generating language description for ui elements. the task is important because missing captions is a major issue for mobile accessibility and addressing the issue can improve accessibility and empower language-based mobile interaction in general. we created a large-scale dataset for this novel task by asking human annotators to create widget captions for a mobile ui corpus via crowdsourcing. we formulate widget captioning as a multimodal captioning task where both structural and image input are available. we experimented with a set of models based on the dataset. the winner configuration-a transformer structural encoder coupled with a resnet cnn-can generate semantically meaningful captions for sparsely labeled elements on the screen, which shows the feasibility of this task and opportunities for future research.