Mapping Natural Language Instructions to Mobile UI Action Sequences (2020)

Abstract: We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it. For full task evaluation, we create PixelHelp, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator. To scale training, we decouple the language and action data by (a) annotating action phrase spans in How-To instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces. We use a Transformer to extract action phrase tuples from long-range natural language instructions. A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions. Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PixelHelp.

## introduction language helps us work together to get things done. people instruct one another to coordinate joint efforts and accomplish tasks involving complex sequences of actions. this takes advantage of the abilities of different members of a speech community, e.g. a child asking a parent for a cup she cannot reach, or a visually impaired individual asking for assistance from a friend. building computational agents able to help in such interactions is an important goal that requires true language grounding in environments where action matters. an important area of language grounding involves tasks like completion of multi-step actions in a graphical user interface conditioned on language instructions @xcite @xcite . these domains matter for accessibility, where language interfaces could help visually impaired individuals perform tasks with open the app drawer. navigate to @xmath0. click add network, and then enter starbucks for ssid. ## problem formulation given an instruction of a multi-step task, @xmath1:@xmath2, t 2 , ..., t n ), where t i is the ith token in instruction i, we want to generate a sequence of automatically executable actions, a 1:m , over a sequence of user interface screens s, with initial screen s 1 and screen transition function s @xmath3, s j-1 ): an action a @xmath4, o j , u j ] consists of an operation r j (e.g. tap or text), the ui object o j that r j is performed on (e.g., a button or an icon), and an additional argument u j needed for o j (e.g. the message entered in the chat box for text or null for operations such as tap). starting from s 1 , executing a sequence of actions @xmath5: each screen s @xmath6,1:|s j | , λ j ] contains a set of ui objects and their structural relationships. c j,1:|s j | = {c j,k | 1 ≤ @xmath7, where |s j | is the number of objects in s j , from which o j is chosen. λ j defines the structural relationship between the objects. this is often a tree structure such as the view hierarchy for an android interface foot_1 (similar to a dom tree for web pages). an instruction i describes (possibly multiple) actions. let āj denote the phrases in i that describes action a j . āj = [r j , ōj , ūj ] represents a tuple of descriptions with each corresponding to a span-a subsequence of tokens-in i. accordingly, ā1:m represents the description tuple sequence that we refer to as ā for brevity. we also define ā as all possible description tuple sequences of i, thus ā ∈ ā. we define â as the most likely description of actions for t 1:n . this defines the action phrase-extraction model, which is then used by the grounding model: ## data the ideal dataset would have natural instructions that have been executed by people using the ui. such data can be collected by having annotators perform tasks according to instructions on a mobile platform, but this is difficult to scale. it requires significant investment to instrument: different versions of apps have different presentation and behaviors, and apps must be installed and configured for each task. due to this, we create a small dataset of this form, pixelhelp, for full task evaluation. for model training at scale, we create two other datasets: androidhowto for action phrase extraction and ricosca for grounding. our datasets are targeted for english. we hope that starting with a high-resource language will pave the way to creating similar capabilities for other languages. ## model architectures equation 7 has two parts. p(â j |â <j , t 1:n ) finds the best phrase tuple that describes the action at the jth step given the instruction token sequence. p(a j |â j , s j ) computes the probability of an executable action a j given the best description of the action, âj , and the screen s j for the jth step. ## experiments our goal is to develop models and datasets to map multi-step instructions into automatically executable actions given the screen information. as such, we use pixelhelp's paired natural instructions and action-screen sequences solely for testing. in addition, we investigate the model quality on phrase tuple extraction tasks, which is a crucial building block for the overall grounding quality foot_2 . ## related work previous work @xcite @xcite investigated approaches for grounding natural language on desktop or web interfaces. @xcite contributed a dataset for mapping natural language instructions to actionable image editing commands in adobe photoshop. our work focuses on a new domain of grounding natural language instructions into executable actions on mobile user interfaces. this requires addressing modeling challenges due to the lack of paired natural language and action data, which we supply by harvesting rich instruction data from the web and synthesizing ui commands based on a large scale android corpus. our work is related to semantic parsing, particularly efforts for generating executable outputs such as sql queries @xcite . it is also broadly related to language grounding in the human-robot interaction literature where human dialog results in robot actions @xcite . our task setting is closely related to work on language-conditioned navigation, where an agent executes an instruction as a sequence of movements @xcite @xcite @xcite . operating user interfaces is similar to navigating the physical world in many ways. a mobile platform consists of millions of apps that each is implemented by different developers independently. though platforms such as android strive to achieve interoperability (e.g., using intent or aidl mechanisms), apps are more often than not built by convention and do not expose programmatic ways for communication. as such, each app is opaque to the outside world and the only way to manipulate it is through its guis. these hurdles while working with a vast array of existing apps are like physical obstacles that cannot be ignored and must be negotiated contextually in their given environment. ## conclusion our work provides an important first step on the challenging problem of grounding natural language instructions to mobile ui actions. our decomposition of the problem means that progress on either can improve full task performance. for example, action span extraction is related to both semantic role labeling @xcite and extraction of multiple facts from text @xcite and could benefit from innovations in span identification and multitask learning. reinforcement learning that has been applied in previous grounding work may help improve out-of-sample prediction for grounding in uis and improve direct grounding from hidden state representations. lastly, our work provides a technical foundation for investigating user experiences in language-based human computer interaction.