UGIF - D ata S et: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI (2024)

Abstract: Help documents are supposed to aid smartphone users in resolving queries such as “How to block calls from unknown numbers?”. However, given a query, identifying the right help document, understanding instructions from the document, and using them to resolve the issue at hand is challenging. The user experience may be enhanced by converting the instructions in the help document to a step-by-step tutorial overlaid on the phone UI. Successful execution of this task requires overcoming research challenges in retrieval, parsing, and grounding in the multilingual-multimodal setting. For example, user queries in one language may have to be matched against instructions in another language, which in turn needs to be grounded in a multimodal UI in yet another language. Moreover, there isn’t any relevant dataset for such a task. In order to bridge this gap, we introduce UGIF-DataSet, a multi-lingual, multi-modal UI grounded dataset for step-by-step task completion on the smartphone, containing 4,184 tasks across 8 languages. The instruction steps in UGIF-DataSet are available only in English, so the challenge involves operations in the cross-modal, cross-lingual setting. We compare the performance of different large language models for this task and find that the end-to-end task completion rate drops from 48% in English to 32% for other languages, demonstrating significant overall headroom for improvement. We are hopeful that UGIF-DataSet and our analysis will aid further research on the important problem of sequential task completion in the multilingual and multimodal setting.

## introduction smartphone users often struggle to navigate the ui to get things done. this problem is particularly acute in developing countries due to varying 1 pronounced with a soft-g: u-jif literacy levels, high cost of phone ownership, etc. @xcite . many of the tasks users struggle with are documented as frequently asked questions (faqs) on support sites foot_0 with step-by-step instructions describing what the user should do on the ui. we explore the problem of harnessing such help documents to create step-by-step tutorials overlaid on the phone ui as an instance of cross-lingual, cross-modal sequential action prediction. to create step-by-step tutorials on the ui using help documents, research challenges in several natural language processing components including retrieval, parsing, and grounding have to be overcome. but no relevant dataset exists for this task in the multilingual setting. we build on prior work in the nlp community in this area @xcite and extend it in the multilingual and multimodal directions. we collect a new multi-lingual, multimodal ui grounded dataset called ugif-dataset to evaluate how well models can predict sequential actions on the phone ui. the dataset consists of 523 how-to queries per language and for each query, step-by-step instructions in english and a sequence of ui screenshots and actions that show how to complete the task. each how-to query and ui sequence is available in 8 languages. an outline of the structure of this dataset is shown in fig. 1 . the tutorial task poses both multi-lingual and multi-modal challenges. many smartphone users are bilingual and ask queries in their native language, but the help documents are often available only in english. hence the need for cross-modal, cross-lingual retrieval. furthermore, users may use a non-english ui / system language which necessitates cross-lingual ui grounding to map the instruction steps in english to ui screens containing different languages. while current multi-modal models have tended to focus on tasks related to a single image, such as caption generation @xcite or grounding the user's command to a ui element on the screen @xcite , the tutorial task introduces another challenge by requiring the model to perform a sequence of actions across ui screens while referencing a help document. finally, since the ui changes often, the help documents are often out-of-date, which introduces the additional difficulty of utilizing potentially unreliable help instructions to complete the task. we propose an initial uni-modal approach that splits this task into retrieval, parsing, and grounding and use existing large language models @xcite as a baseline to explore the challenges in this task and estimate the headroom available for improvement. the contributions of this work are as follows: similar to that, worldofbits is an open platform for training web navigation agents @xcite . in our work, we take the lat-ter approach of exposing deeper functionality of a few popular apps by relying on help articles in the android support site. we chose this because new users often ask goal oriented questions that require greater knowledge about how to navigate a particular app. moreover, app developers often provide faqs with common tasks in mind, so we can exploit the support pages to create ui grounded tutorials for new users. pre-training for ui tasks: in the past few years, there has been a paradigm shift in deep learning towards pre-training on broad unlabelled datasets and fine-tuning on task specific data. @xcite ; @xcite pre-train a transformer model on a large number of screenshots obtained by crawling apps in smartphones in a manner similar to web crawling. since our focus is on multilingual ui screens, we chose to use the pre-trained labse @xcite for ui grounding, but utilizing broad ui data will be critical for future improvements. large language models: large language models (llms) pre-trained on large corpora of text scraped from the web have shown remarkable fewshot generalization capability @xcite . we employ llms for parsing help articles but not for ui grounding since we prefer to do it on-device for privacy reasons. language grounding in human-robot interaction: language guided robot actions for humanrobot interaction @xcite is a broadly related problem. however, taking actions on real robots is much more complex with uncertain outcomes, whereas precise actions can be performed on the ui with near certainty. as a result, the difficulty with ui grounded interactions is less about sensing and actuation and more about understanding user intent and navigating the app by understanding its structure using external resources such as support pages. icon and widget captioning: although android allows developers to provide content description for images, not all app developers do so. to support a wide range of apps, it becomes necessary to recognize icons and widgets @xcite . in our work, all the apps provide the necessary description, so icon captioning is not necessary. ## ugif-dataset: a new multilingual multimodal ui-grounded instruction following dataset to build and evaluate an android ui navigation agent that can teach users how to use the ui, we collect a new multi-lingual, multi-modal ui grounded dataset called ugif-dataset. it is a corpus of howto queries in text and speech in multiple languages, instruction steps for each tutorial paired with sequences of ui screens and actions as the tutorial is completed by human annotators on android devices (fig. 1 ). the pixel help support pages provide step-bystep instructions for performing common tasks on android. this is an example task: "how to block unknown numbers?" for which the instruction text is "1. open your phone app 2. tap more. 3. tap settings and then blocked numbers. 4. turn on unknown". we crawl the android support site and extract the tutorial steps using simple rules that look for ordered lists under a header. annotators translate and speak out loud the how-to query. they also parse the tutorial steps to a sequence of macros in we used an internal platform for crowd-sourcing annotations from annotators in india, kenya, and mexico. annotators were screened by asking qualifying questions with multiple-choice answers. only those who successfully answered the qualifying questions were allowed to participate in the data collection. the qualifying questions tested linguistic capability: simple questions to gauge whether they can understand written content and speech in the target language. there was no android ui or task specific training involved. the task description explained the purpose of this data collection effort and how it would be used for research. to protect the privacy of the annotators, the annotator id is not included in this dataset release. the task design was reviewed by privacy, ethics, and legal committees. the price set for each task was in compliance with local laws. ## experiments the ugif-dataset dataset contains manually annotated oracle parses (macro sequences) for each how-to instruction text. we measure parsing accuracy by looking for an exact match between the generated parses and the oracle parses. the dataset also contains manually annotated screen-action sequences for the entire how-to, but it does not have such sequences for each macro. so, to evaluate the grounding model, we consider the end-to-end task completion success rate. although it is possible to complete each task in more than one way, we want to follow the how-to in-struction text exactly, so we consider a task to be completed successfully only if the entire sequence of actions predicted by the model exactly matches the sequence of actions taken by the annotator. ## conclusion we proposed helping new smartphone users by showing them how to perform tasks on the ui based on voice queries. we evaluated existing language and sentence similarity models for the task of retrieving and executing how-to instructions on the ui where the ui language potentially differs from the language used in the instruction text. the models we build for this task must be capable of adapting to minor variations in the ui as the newer versions of the app are frequently released and instructions become outdated. multilingual uis pose the challenge of having to simultaneously work with multiple languages in a single ui screen since app developers may not have provided translations for all ui elements. finally, our evaluation of current pre-trained models suggests that there is significant room for improvement and that a multimodal language-ui foundation model could lead to substantial gains. the user interface evolves much more frequently than natural language or images. as a result, the dataset and models trained on it may need adaptations, which we leave as a topic of future exploration. since we have captured ui screens at a particular point in time, we were unable to quantify the reduction in task completion rate due to ui drift or investigate methods specifically aimed at addressing such ui changes. an important future direction is to capture such ui changes as apps evolve over time and investigate how well models generalize to these changes. our dataset contains only one speech sample per query in each language, so the diversity of speech samples is limited. moreover, the speech samples were crowd-sourced by asking annotators to speak out loud the how-to query in the title of the faq page, so this may not match how users might ask queries with the same intent without being prompted with the faq page. all the instructions have been scraped from the google support site, so our evaluation of parsing does not cover instruction text on forums and other support sites. furthermore, all the ui captures in our dataset start at the home screen, but it would be desirable to also evaluate ui grounding from arbitrary starting points.