Referring to Screen Texts with Voice Assistants (2023)

Abstract: Voice assistants help users make phone calls, send messages, create events, navigate and do a lot more. However assistants have limited capacity to understand their users’ context. In this work, we aim to take a step in this direction. Our work dives into a new experience for users to refer to phone numbers, addresses, email addresses, urls, and dates on their phone screens. We focus on reference understanding, which is particularly interesting when, similar to visual grounding, there are multiple similar texts on screen. We collect a dataset and propose a lightweight general purpose model for this novel experience. Since consuming pixels directly is expensive, our system is designed to rely only on text extracted from the UI. Our model is modular, offering flexibility, better interpretability and efficient run time memory.

## introduction with the advent of internet and smartphones, the world came to our fingertips. and with the emergence of voice assistants (vas), everything became even more accessible. vas have become pervasive in the smartphones as they offer natural means of communication to the user. they able a user to perform tasks faster with natural language instead of several taps, app switches, scrolls and typing. however, they are limited in their ability to understand the user's context. let us look at an example. in fig. 1 , a user wants to share a number from a webpage to a friend. they might do either of the following: we conduct a user study to explore how users would make requests involving screen elements. participants are shown screenshots, each containing multiple entities of a category (eg. 3 phone numbers), and asked to type requests for a va to act on one of them. the study reveals that a majority of users (57%) prefer to use references like "send that office number to tim' instead of repeating the full text. for supporting such experiences, voice assistants need to resolve the references. in this work, we focus on such reference resolution. specifically, we consider requests referring to phone numbers, addresses, email addresses, urls, date/time. we figure 1 : suppose a user wants to share a number on their screen. we aim to support this in a natural and succinct way by enabling users to refer to screen elements in interactions with voice assistants. choose these actionable text since, ∼50% of screen elements are texts @xcite , and these categories are commonly acted upon. users often call or message numbers, share contact details, navigate to addresses. to understand and evaluate the task, we collect screenref, a dataset of 14k requests, with references to actionable text or entities. screenref contains two collections. first, descriptive data, which is based off screens with multiple similar entities to get descriptive references like 'call the apple business manager number'. this is similar to how visual grounding datasets @xcite focus on images with multiple objects of the same category to get challenging references. second, category level data, includes simpler references to a category without disambiguating within a category, eg. 'call this number'. these are described in sec. 4. with the purpose of deploying in real world, it is critical to design solutions with low latency. to this end, we design screen reference resolver, or srr, a modular attention based architecture for reference understanding. we focus on privacy, hence our model is lightweight and executable on-device. our network re-uses existing signals available from upstream including request embedding, and text scraped from the ui. we also discuss a heuristicbased baseline (designed for quick prototyping). overall, our main contributions are: ## related work grounding to ui elements. past works have explored mapping natural language commands to ui elements for chrome web pages @xcite , grounding executable actions for ui navigation @xcite and user interaction @xcite . these works primarily focus on navigational commands, thus target buttons, links and input boxes. our goal is to explore screen referencing capability for common va tasks, thus we target 'actionable text entities' like phone numbers. hsiao et al. (2022) propose screenqa with questions about ui elements including text, which could also benefit from ui grounding. wang et al. (2022); rozanova et al. (2021) investigate llm abilities for ui grounding. li et al. (2021); li and li (2022 ) use vision and language transformers for the task. however, we only use the screen texts and no pixels directly. our solution design focuses on low latency, less memory and privacy-preserved inference that can be run on device. ## user study to understand how users would make requests about entities on the screen, we conduct a user study on @xcite . we use 4 screens for each actionable text category (phone numbers, emails, addresses, and urls), and each screen has 3 instances of the category (eg. 4 screens with 3 phone numbers each). a total of 300 participants are selected from across us, balanced for va usage, gender and english as first or other language. overall, 4800 typed requests were collected. the responses were reviewed by two researchers. using heuristics, three common types of requests surfaced: 1. full text: "call 1-866-902-7144" 2. labels: using text other than full entity text "directions to the one in portland" 3. ordinals: "send the third email address". the data shows a heavy preference towards the first two (fig. 2 ). intuitively, when browsing information, the eyes are often scanning for a topic of interest. for instance, "i need to call support", explaining the label based requests. our hypothesis for the high use of full text is that they didn't want to rely on va's ability to understand the context. within references, using the text in or around the entity is common and the position is used sometimes. note that our study was performed on a limited set of users and for a limited set of screens, but we uncover interesting patterns on how users might request actions on screen texts. it is important to keep in mind that speaking full texts could be cumbersome, unnatural and have speech recognition errors, especially for entities like urls. ## task and dataset given a screen s (with ocr texts t), text entities e 1 , ..., e k and a request r, the task is to select the entity(entities) @xmath0, ..., e k being referred to in r. we collect screenref, a collection of requests to voice assistants with references to actionable text categories on screen (phone number, address, email address, url, date/time). due to privacy concerns on sharing a dataset with extracted phone numbers/emails from web pages, we are unable to share the dataset but we discuss the collection protocol in detail (see samples in fig. 4 and annotation guidelines in appendix a). we collect full requests, not just reference phrases, since words outside of the explicit reference phrase may hint at the targeted entity. for instance, call this has the reference this which is ambiguous, however the request can be understood as referring to a phone number. 20 annotators are recruited for our data collections. we first started with a simple collection protocol. after extracting entities of our interest using data detectors for a list of web pages, we show a web page screen with one highlighted entity and ask graders to provide a request referring to that entity. this would get us a dataset of requests referring to screen entities and their referred entity. however, this ran into major issues. first, annotators would often miss other similar entities on the screen and provide requests which are ambiguous, eg. "call this number", when there is more than 1 number on the screen and only 1 of them is highlighted, thereby resulting in an incorrect sample for the resolution task. second, there were a large number of duplicate requests (>40%). this may happen due to several screens may have one entity and thereby annotators may use simple references. other issues included the need of screens with entities to collect any data, lack of representation of different reference types within the collected data and lack of awareness of ambiguous requests. the quality and efficiency concerns led us to develop a new protocol in the form of descriptive and category level data collections. within descriptive collection, we use a similar screen based collection technique. however we restrict to screens with more than one instance of a category in order to collect challenging and diverse requests (similar to visual grounding datasets like refcoco). alongside the target entity, we highlight all entities of that category to reduce chances of erroneous ambiguous requests. within category level collection, we do not use screens and the focus is on unique diverse requests with simple references. this split addresses the issues described above leading to more efficient collection and better quality datasets. descriptive data collection. though we aim to support references on all apps on phone, this collection is carried out with web pages due to their varied layouts and ease of access. for a list of top visited web pages, we extract texts by ui scraping and get text categories using data detectors. this is similar to running object detectors in open scenes. in order to get challenging references, we only keep screens with more than one entity from a category (eg. 2 urls). one entity on the screen is highlighted as target and users are asked to provide requests for that (fig. 7b ). guidelines provided in a.2. for quality check, we run a verification to confirm the requests are unambiguous: three independent annotators are shown the screen and the collected request and asked which entity from the screen is the request referring to. samples where at least 2/3 annotators agree are kept, leading to ∼6% data drop. category-level data collection. this collection targets simple references for a category ("phone number -call that number", "url -open it"). during screen mining, we observe that a lot of the screens have only one entity from a category. in these cases, users may prefer succinct simple references instead of descriptive ones. note that these references do not use the screen layout. hence, we design this collection independent of screens. this gives a simpler collection scheme that allows us to scale to new categories and/or locales more quickly, with reduced time and cost. we show a category and ask annotators to give requests, assuming that entity is on their screen. the collection is carried out on shared spreadsheets, one sheet per category (fig. 7a ) in order to avoid duplicate requests across annotators. annotators are given automatic instant feedback by count_unique to encourage variations. through pilot annotation projects, we recognize several constraints to ensure that the uniqueness is not from spurious modifications, which are also added to the guidelines. detailed grading guidelines provided in a.1. for verification, 3 independent annotators are shown a request and asked to mark all categories it could refer to. this also gives annotated multi-label samples i.e. requests that are category ambiguous: "take me there" could be referring to a url or an address. requests with majority agreement are kept. after the requests are collected, dummy entities, one of each scoped category, are added to each request to form a data sample. in a way, this makes the dataset more complete and challenging than real screens which may include only a subset of the entity categories. (fig. 4 ). 5 models 5.1 heuristic-based baseline this is designed for quick prototyping and development without much training data. we define a set of hand-crafted rules using keywords from a subset of the training data. the rules are applied in sequence: 1. phrase-match. look for synonyms or verbs or apps in the request that indicate the target category (like 'number', 'call' indicate phone number, 'navigate', 'maps' indicate address). 2. location-match. regex match to find positional or ordinal reference in request, sort candidates by coordinates and pick the entity at the mentioned position. 3. label-match. locate the text on screen that ## conclusion we explore a new user experience of executing actions on screen elements with voice assistants. to make interactions more natural, we explore the use of references. an important decision was what ui elements to support. we decided to use texts that are most commonly used for task oriented dialogue and, commonly present on phone screens and easy to classify. we collected a dataset of requests and proposed solutions to understand references. this is a step towards making voice assistants more context aware, but there is a lot more context. we hope that our work will motivate further research towards this goal, and towards semantic visual text referencing.