Towards Better Semantic Understanding of Mobile Interfaces (2022)

Abstract: Improving the accessibility and automation capabilities of mobile devices can have a significant positive impact on the daily lives of countless users. To stimulate research in this direction, we release a human-annotated dataset with approximately 500k unique annotations aimed at increasing the understanding of the functionality of UI elements. This dataset augments images and view hierarchies from RICO, a large dataset of mobile UIs, with annotations for icons based on their shapes and semantics, and associations between different elements and their corresponding text labels, resulting in a significant increase in the number of UI elements and the categories assigned to them. We also release models using image-only and multimodal inputs; we experiment with various architectures and study the benefits of using multimodal inputs on the new dataset. Our models demonstrate strong performance on an evaluation set of unseen apps, indicating their generalizability to newer screens. These models, combined with the new dataset, can enable innovative functionalities like referring to UI elements by their labels, improved coverage and better semantics for icons etc., which would go a long way in making UIs more usable for everyone.

## introduction mobile devices like phones and tablets have become ubiquitous and indispensable to carry out our daily activities. it is not an exaggeration to say that usage of mobile devices is becoming a requirement for full participation in society. recent reports from the who and others @xcite estimate that around 2.2 billion people across the world have some form of vision impairment, out of which 36 million people are blind. accessibility of mobile *equal contribution, correspondence: {srinivasksun,mariawang}@google.com devices is necessary for these visually impaired users to carry out their daily tasks and is an important tool for their social integration @xcite . accessibility of mobile apps has improved significantly over the past few years aided by developments on two main fronts: firstly, the development of screen readers, like voiceover @xcite on ios and talkback @xcite on android, enable visually impaired users to control their phone in an eyes-free manner. secondly, development tools and standards to enhance accessibility, such as the accessibility guidelines for ios and android @xcite , android accessibility scanner (accessibility, 2021b), and ios accessibility inspector @xcite , have helped developers identify and fix accessibility issues for applications. among most of these utilities, the main source of accessibility data is the accessibility labels @xcite provided by app developers. these labels are specified as attributes on a structured representation such as view hierarchy, for the different ui elements on the screen and are available to screen readers @xcite . despite the growth in accessibility tools, recent studies @xcite @xcite have found that even the most widely used apps have large gaps in accessibility. for instance, a study by @xcite of more than 7k apps and 279k screens revealed that around 77% of the apps and 60% of the screens had at least one element without explicit labels. @xcite found that, in a population of 10k apps, 53% of the image button elements were missing labels. in this paper, we attempt to encourage further research into improving mobile device accessibility and increasing device automation by releasing an enhanced version of the rico dataset @xcite with high-quality human annotations aimed at semantic understanding of various ui elements. firstly, following a study by @xcite where missing labels for image button instances was found to be the primary accessibility barrier, we focus on creating annotations useful for identifying icons. in particular, we annotated the most frequent 77 classes of icons based on their appearance. we refer to this task as the icon shape task. secondly, we identified icon shapes which can have multiple semantic meanings and annotated each such icon with its semantic label. this task is called icon semantics. some examples of such icons can be seen in figure 1b . finally, we annotate ui elements, like icons, text inputs, checkboxes etc., and associate them with their text labels. these associations can help us identify meaningful labels for the long tail of icons and ui elements not present in our schema, but having a textual label associated with them. we refer to this task as the label association task. the main contributions of this paper are as follows: ## datasets, taxonomy and annotation to enable accessible hands-free experience for mobile users, it is necessary for the system to understand the functionality of the different screen ui elements. for learning data driven models to enable these functionalities, we use the rico dataset @xcite . rico @xmath0, each with a screenshot and information regarding the structure of the ui in the form of a view hierarchy (vh). besides the bounding boxes of the different ui elements, the vh contains useful attributes like the content description and resource id which provide information regarding the functionality of the different ui elements. however, @xcite found that only 35% of the unique screens in rico contain a matching view hierarchy and screenshot. in the next few sections, we describe how we overcome the mismtach issue and describe the different type of annotations. ## baseline models we conducted several experiments to investigate the effectiveness of various deep learning approaches for solving the tasks presented in section 3. the overall goal of these experiments is to: 1) provide good baseline models to be used for image-only and image+vh settings 2) study the effect of using multi-modal inputs v/s only the screenshot for these tasks. ## applications and future work this dataset and models built on it to predict the icon and text association labels can be used to improve the label coverage for various accessibility applications like voiceover @xcite , talk-back (accessibility, 2021e), voice access (accessibility, 2022) etc. in addition, these labels can be used to improve the accessibility labels for various other platforms like web browsers and desktop applications. the models can also be used to automatically suggest accessibility labels for ui elements based on their appearance in various developer platforms (xcode, 2022; androidstudio, 2021) so that developers can improve the accessibility of their apps easily. along with improving accessibility, we believe this dataset is a step towards enabling new features like voice control, screen summarization and others. the baseline models presented in this paper can be improved in a number of ways like training multitask models for a single model to output the different labels, bridge the gap between the models which use only the image and models which use view hierarchy, improve the ability of multimodal models to handle missing view hierarchy elements and their attributes. other directions of research include inferring labels for the long tail icon classes using the icon class annotations from the label association task, inferring semantic labels for general ui elements. this supervised data can also be used to improve the performance of self-supervised methods for ui elements like actionbert @xcite , uibert @xcite etc. ## conclusions in this paper, we presented an enhanced version of the rico dataset with three new sets of annotations aimed at improving the semantic understanding of mobile screens, namely icon shape, icon semantics, and label association. we outlined the benefits of human annotated data over automatically labeled data and released strong baseline models using image and view hierarchy for each of these tasks. our dataset, benchmark models and experiments lay the groundwork for future research on build-ing better models for semantic understanding of uis. we observe that using pre-trained models and view hierarchy attributes is a promising direction for improving these models. these models can be combined with other techniques like heuristic rules to infer the multitude of labels useful for driving improvements in accessibility and automation of mobile devices.