Tell Me What’s Next: Textual Foresight for Generic UI Representations (2024)

Abstract: Mobile app user interfaces (UIs) are rich with action, text, structure, and image content that can be utilized to learn generic UI representations for tasks like automating user commands, summarizing content, and evaluating the accessibility of user interfaces. Prior work has learned strong visual representations with local or global captioning losses, but fails to retain both granularities.To combat this, we propose Textual Foresight, a novel pretraining objective for learning UI screen representations. Textual Foresight generates global text descriptions of future UI states given a current UI and local action taken. Our approach requires joint reasoning over elements and entire screens, resulting in improved UI features: on generation tasks, UI agents trained with Textual Foresight outperform state-of-the-art by 2% with 28x fewer images. We train with our newly constructed mobile app dataset, OpenApp, which results in the first public dataset for app UI representation learning. OpenApp enables new baselines, and we find Textual Foresight improves average task performance over them by 5.7% while having access to 2x less data.

## introduction people use mobile apps every day to browse news articles, shop online, book appointments, and learn from educational platforms @xcite . ai agents can help to perform these real-life tasks for those who cannot or prefer not to view or touch the app screen (e.g., users who are blind, low-vision, or busy driving) @xcite . to build such ai models, a key question is which modalities should be used to represent the app ui, as it consists of not only the rendered screen, but also metadata, text, and structural features (i.e., the underlying app view hierarchy). recent work spotlight learns ui features with only the rendered screen image @xcite , as the view hierarchy is not always available, and while both element captioning and textual foresight pretraining aim to preserve the semantics of individual ui objects, textual foresight also requires understanding global ui semantics of the current screen and how an action on the ui will change it, as the objective is to generate the global description of the following screen. we highlight in red the ui object associated with the input bounding box coordinates. when it is, it often contains generic, noisy, or missing fields @xcite . spotlight proposed ui representation learning via element captioning, and is state-of-the-art on four downstream ui tasks. while element captioning avoids the disadvantages of other ui modalities, it only enforces local ui understanding. as shown in figure 1 (left), this objective trains a model to map an image and bounding box coordinates to an element-level caption like "options." however "options" is a limited representation of what this element can do, as it lacks context from the global ui screen or what action it affords. if we enlarge the visual context to the entire screen, we see that it contains different songs in a streaming application like spotify. yet only when seeing the screen that appears upon clicking the "options" element, figure 1 (right), we finally understand that it provides the means to like, hide, or share a particular song. our goal is to better balance local element and global screen features, and we find that ui actions can serve as the bridge between them. an action performed on a ui informs the semantics of the next ui state. following this intuition, we propose textual foresight: a representation learning objective that generates global screen captions of a future ui, given a current ui image and a localized action. this task requires understanding both the local semantics (options icon) and global semantics (a spotify music playlist) of the current input ui to be able to decode the caption "song information and options for playing, saving, sharing, reporting explicit content, and viewing credits." it also benefits from (state, action) examples, implicitly teaching element affordance. to study textual foresight, we build openapp, the first publicly available dataset for representation learning in apps. state-of-the-art spotlight did not make their pretraining data available, and does not benchmark on a fully open-source evaluate suite, either. we curate openapp with multiple element-and screen-level caption sets, which we use to reproduce spotlight and train other baselines like screen captioning which have never been studied before. we design our framework on top of blip-2 @xcite , making all code publicly available, unlike spotlight, which also did not open source model code nor checkpoints. our experiments show that textual foresight is able to better balance the granularity of features learned: it reaches the best average performance for screen summarization and element captioning tasks, which require global and local ui features, respectively. importantly, textual foresight reaches better performance while having 28x less pretraining data than spotlight, and 2x less than our new baselines. textual foresight consistently performs best among our open-source baselines, resulting in a 5.7% average task performance boost. in summary, our contributions include: • a novel pretraining objective, textual foresight, which learns ui representations by describing future ui states given the current screen and a localized action. textual foresight outperforms sota spotlight for generation-style tasks with 28x less data. ## related work while there are several prior methods for learning ui representations, all either use proprietary data and/or evaluate on different tasks, making downstream comparison challenging. figure 2 compares textual foresight to actionbert @xcite , screen2vec @xcite , uibert @xcite , and spotlight @xcite . we compare the type of loss (predictive or generative) and if the loss utilizes action data from the ui. as we see in the upper right quadrant, textual foresight is the first generation style loss to incorporate action. textual foresight and spotlight are bolded, as they only input the screen image to represent the ui. in addition to textual foresight and element captioning, global image captioning has been used to learn representations of natural rgb images. e.g., it is one loss within blip-2 @xcite , which is sota on visual question answering, image-text retrieval, and captioning. it has also been used to learn features for vision-only tasks, matching or outperforming sota for image classification, object detection, and instance segmentation, while using 10x fewer images during training @xcite . image captioning has never before been studied as a method to learn ui representations due to a lack of available screen caption data. we also consider how foresight has been used in prior work. visual foresight was first introduced to improve robot motion planning @xcite and has since been incorporated in numerous works in robotics @xcite , reinforcement learning @xcite @xcite and vision language navigation @xcite . differing from textual foresight, which predicts language descriptions of future states, these prior works predict raw images @xcite or intermediary visual features @xcite @xcite . finally, we note that there are several prior works on multimodal ui tasks, datasets, and pretraining approaches in the context of webpage understanding. these works have studied multimodal web agents @xcite , mul-timodal web summarization @xcite , and web captioning @xcite . ## ui representation learning with textual foresight we aim to learn strong generic ui representations that can be used across many downstream ui tasks. given a ui screen image s t , the goal of textual foresight is to describe what follows from taking action a t on it. by training a vision-language model (vlm) with textual foresight, a single loss can encourage the visual representations to retain both local and global features over the ui screen. in figure 3 , we show how we can learn meaningful features over the input screen image by asking a foresight question. we input a single ui from a longer action sequence, like the chrome browser state with options opened, and ask what is expected from clicking on "new tab." visually understanding the new tab element in isolation does not tell us much about the current screen or how interacting with the element would be useful. yet to be able to describe the future ui as "a search engine app with various popular website shorted and suggested articles," it requires learning a ui representation that captures not only the semantics of "new tab," but also the global visual context that the input ui contained a search engine result screen. in section 3.1 we define the training loss for textual foresight, and then detail model pretraining and finetuning in section 3.2 and 3.3, respectively. ## openapp dataset as shown in figure 3 , textual foresight requires mobile app action sequences. in addition to needing data for our new method, other baselines have never been explored due to data limitations (e.g., large scale screen captioning data did not exist) or only studied in a proprietary setting (e.g., element captioning data used in spotlight). to curate pretraining data for textual foresight and important baselines, we combine and generate new data for existing app datasets motif @xcite , one snapshot from the longitudinal study by @xcite , and android in the wild @xcite . we refer to the merged data source that we further annotate and post process as openapp. the raw openapp data consists of app action sequences, with each time step having an action annotation and a corresponding ui screenshot and view hierarchy; we now detail the new annotations and data post-processing. appendix b contains examples of each resulting caption set, additional processing details, and a discussion on potential dataset noise. ## experimental setup we now describe the new baselines made possible with the openapp dataset and pretraining and finetuning experimental settings. ## results we now report results for generative tasks (screen and element captioning) and prediction tasks (tappability classification and language grounding). ## conclusion in this work we have proposed using ui actions as the bridge between local element semantics and global screen context. specifically, we introduced a new pretraining objective, textual foresight, which trains a model to describe a future screen image given an action taken on the current viewed state. to train our new model we contribute a new dataset, openapp, which contains screen and element level captions for 5.7m app images that can be used for training several baselines. we are the first to provide an open-source app dataset for ui representation learning and evaluate on a standardized downstream benchmark. our textual foresight approach can use only a subset of this data and on average outperforms not only our open-source benchmarks, but also prior state-of-the-art method spotlight on generation tasks, while using 2x less data than open-source baselines, and 28x less data than prior state-of-the-art.