UIN av: A Practical Approach to Train On-Device Automation Agents (2024)

Abstract: Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired. Prior automation systems do not produce generalizable models while AI-based automation agents work reliably only in simple, hand-crafted applications or incur high computation costs. We propose UINav, a demonstration-based approach to train automation agents that fit mobile devices, yet achieving high success rates with modest numbers of demonstrations. To reduce the demonstration overhead, UINav uses a referee model that provides users with immediate feedback on tasks where the agent fails, and automatically augments human demonstrations to increase diversity in training data. Our evaluation shows that with only 10 demonstrations can achieve 70% accuracy, and that with enough demonstrations it can surpass 90% accuracy.

## introduction the next frontier in artificial intelligence is agents that autonomously operate computers as humans do. instructed by users in natural language, these agents are especially valuable when their users have visual or motor disabilities or when they are situationally impaired (e.g., driving, cooking). we are particularly interested in agents that can execute human tasks by interacting directly with the user interface (ui) of a running application. these socalled ui automation agents @xcite @xcite can scale well to support a myriad of tasks because they do not depend on third-party apis. existing approaches to ui automation range from ui scripting to ai-based agents. ui scripts can work reliably, but they involve coding or manual demonstrations @xcite ., * work done as an intern at google research. a practical approach to ui automation requires trading between accuracy, generalizability and computational costs. we find a sweet spot between these three properties, and propose uinav, a demonstration-based system designed to produce lightweight neural agents that can run on mobile devices while yielding good success rates. as in prior work, uinav needs to address the challenge of how to achieve good success rates with fewer demonstrations. we observe that the demonstrations required to achieve good performance differs widely across tasks and environments. if the environment is relatively static even a handful of demonstrations is sufficient; for tasks that must work across many different uis more demonstrations are needed. when collecting demonstrations, uinav provide users with immediate feedback on which tasks are failing and may benefit from additional demonstrations, and which are satisfactory. it does so through a referee model which is trained with the same set of demonstrations used to train the automation agent, but with a different goal: predicting whether a task is successfully completed (rather than predicting which ui action to perform). another challenge uinav addresses is how to increase the robustness of automation agents against system delays and changes in the ui. it does so through three key techniques. first, every ui action is executed as a small program composed of lower-level operations with status checks. these programs, referred to as macro actions, abstract the system-specific details thus greatly reducing the agent's state space and therefore the number of required demonstrations. second, uinav adopts demonstration augmentation where human demonstrations are augmented by randomizing non-critical ui elements to increase their diversity. finally, through utterance masking variable sub-strings in utterances are abstracted out. we develop uinav using an internal dataset of 40+ tasks and test is on actual android phones. we also evaluate it on a public dataset, where uinav outperforms various baselines and demonstrates generalizability. overall, we make the following contributions: (i) a practical system to build ui automation agents that achieve near perfect success rates on previously seen tasks and that can be deployed to mobile devices; (ii) an error-driven process to collect demonstrations paired with augmentation techniques and macro actions; and (iii) a comprehensive evaluation demonstrating uinav's advantages over state-of-the-art systems. ## why is ui automation hard? we study the problem of how a ui automation system can generalize to new execution environments, including different apps and different tasks, without requiring an excessive number of demonstrations. to illustrate the challenges we use an apparently simple task, search, i.e., operating the search bar of an app. two aspects make this task challenging. search is a universal task that must work across a myriad of apps where search bars can take many different formats. some search bars require the user to type some keywords and then click an icon (typically on the right hand-side); others, as the user types, automatically display search results which can be directly opened; some others have an additional field (e.g., a category) which must be set beforehand; there are also search bars that are hidden and reveal only upon clicking on an icon; etc. the second axis of complexity regards the agent's start state. when an agent is requested to search in a specific app, the user's device screen may not display the target app or may display it in a page (state) without any search functionality. the agent must first understand how to navigate to the state offering the search function, which may involve navigating back, launching a different app, or dismissing welcome screens and ads. even when the environment already shows the desired search widget, its state may need to be reset, e.g., by deleting search terms previously entered (see the youtube example in fig. 6 in the appendix). in general, in a real environment, an agent is exposed to many different screen conditions caused by a combination of factors: different apps, dynamic app content, previous interactions, layout variance across devices, ui changes across app/os versions, ads and notifications, etc. an agent needs to ignore irrelevant ui elements and navigate to relevant states. one way to tackle this variability is through more demonstrations, but with obvious overheads. uinav's first contribution is to adopt an error-driven process to collect demonstrations ( §4.2). its second contribution is to amplify the learning brought by each demonstration by automated augmentation ( §5). finally, to address variability issues due to system delays, rather than relying on demonstrations uinav takes a programmatic approach by introducing macro actions ( §5). ## system design fig. 1 shows the high-level architecture of uinav. given a task represented by a natural language utterance and an observation of the device state (i.e., a representation of what is currently displayed on the screen), a neural network-backed agent responds with its choice of action to complete the task. the predicted action is executed by the environment by interacting with a device's system (an emulator or a real phone). then, the agent is provided with a new observation describing the new state and a new action is predicted. this setup is similar to that of a reinforcement learning agent, but uinav also includes a second agent called referee, which is responsible for judging the completion status of a task (episode) at each time step. the development of uinav agents (left of fig. 1 ) involves first collecting human demonstrations for some target tasks, then training the neural networks of the agent ( §4.1) and referee ( §4.2), and finally evaluating them on the device. failures of either the agent or the referee are recorded and used to guide the collection of new demonstrations to be used in the next round of training. the development loops over these steps until no more errors of either the agent or the referee are found. ## increasing robustness and efficiency we have described how uinav helps developers balance accuracy and number of demonstrations. next, we describe the techniques that increase the agent robustness in the face of system delays, ui changes, and variations in task descriptions. action validation and macro actions. controlling uis of an actual device involves dealing with various system issues. there are unavoidable delays between the time a state is collected from a device and when a predicted action is ready to be performed. screens can also be slow at loading or updating, hence an agent needs to wait for them to stabilize. these delays are particularly noticeable on a mobile device. to deal with these issues, rather than modeling this variability through more demonstrations, we take various programmatic measures. first, before executing an action, uinav validates it by checking whether a referenced ui element is still on the current screen and if so, whether it has changed. if the action is not applicable anymore, it requests a new prediction. second, every action is executed as a small program that is composed of lower level operations with status checks. such a program is referred to as macro. each macro is implemented following a state transition graph and it is atomic so that while a macro is running the agent stays idle and changes to the screen are not visible to it. an example of macro action is focus_and_type which comprises 4 low-level actions: clicking the input field to obtain focus, waiting for the blinking cursor to appear, typing the text in the field, and (optionally) pressing enter. see §a.4 for more details. demonstration augmentation. to further limit the number of required demonstrations and amplify the learning brought by each one, uinav also augments the collected demonstrations by randomizing the attributes of randomly-selected, non-critical ui elements. this teaches the agent which elements may be safely ignored, and ultimately makes it more tolerant to ui changes. non-critical ui elements have their attributes modified with a predefined probability by either (i) replacing the embedding of their text labels with random vectors, or (ii) by adding random offsets to the four scalars of their bounding boxes, which is equivalent to randomizing both the element's position and size. despite its simplicity, demo augmentation is highly effective at improving uinav's performance (see ## system evaluation we built uinav for android. both the agent and referee are implemented in tensorflow. the agent model has 320k parameters and its tflite version occupies 1.3mb, while the referee has 430k parameters and it is 1.8mb large. for text encoding we use smallbert @xcite and convert it to a 17.6mb tflite model. no quantization is applied during the conversion (more implementation details in §a.5). as shown in ## limitations to limit the number of required demonstrations, the uinav agent makes decisions based only on the contents of the current screen and does not utilize information from previous screens. however, if a task truly requires an agent to remember previous states or actions, then the current architecture of the agent model will fail. our assumption is that a welldesigned ui often presents all the information that is needed for successful human interaction on the current screen. the accuracy of our memory-less agents proves that this is the case for the tasks tested so far. for tasks or uis that require memory, the uinav agent model can be enhanced with memory through either a recurrent neural network or by padding previous states in its input. our approach depends on ui elements for both representing features of screens as well as defining actions. it will not work if a screen representation fails to capture critical ui elements. this can happen also when accessibility trees miss critical nodes because content embedded in webviews and canvas is generally not captured. ## conclusions we presented a demonstration-based system for building small and fast ui automation agents that are suitable for mobile devices. our approach requires small human effort and no coding skills. with modest numbers of demonstrations uinav agents achieve near perfect success rate on previously-seen tasks and with more effort they can generalize well to new tasks and applications. and lijuan wang. 2023. gpt-4v in wonderland: large multimodal models for zero-shot smartphone gui navigation. xiaoyi zhang, lilian de greef, amanda swearngin, samuel white, kyle murray, lisa yu, qi shan, jeffrey nichols, jason wu, chris fleizach, aaron everitt, and jeffrey p bigham. 2021. screen recognition: creating accessibility metadata for mobile applications from pixels. in proc. of the 2021 chi conference on human factors in computing systems, chi '21. boyuan zheng, boyu gou, jihyung kil, huan sun, and yu su. 2024. gpt-4v(ision) is a generalist web agent, if grounded. arxiv preprint arxiv:2401.01614.