META - GUI : Towards Multi-modal Conversational Agents on Mobile GUI (2022)

Abstract: Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task. However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs. In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD). A GUI-TOD system can directly perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. Furthermore, we release META-GUI, a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. We also propose a multi-model action prediction and response model, which show promising results on META-GUI. The dataset, codes and leaderboard are publicly available.

## introduction recent years have witnessed the rapid development of task-oriented dialogue systems @xcite @xcite . they have been widely applied to customer support, booking system and especially intelligent personal assistant. these task-oriented dialogue systems work in a similar pipeline: firstly identify the user intent, then extract necessary information by the process of slot-filling. after getting enough information for the task, the agent will call the backend apis (provided by app developers) to fetch infor-figure 1 : an example of the gui-based task-oriented dialogue system(gui-tod). the action executor will execute tasks on gui and the system will generate a response based on the execution result. ## meta-gui creation our dataset consists of two kinds of data: dialogues and gui operation traces. in each dialogue, user would ask the agent to complete a certain task through multi-turn interaction. our tasks involve six different domains: weather, calendar, search, taxi, hotel and restaurant. in this paper, we consider apps that accomplish the same kind of tasks to be in the same domain. to enhance the diversity of our dataset, we use multiple apps from the calendar and weather domains. the details of apps are listed in appendix a. ## model design the overview of our system is illustrated in figure 5 . it's composed of four components: encoder, image feature extractor, multi-modal information fusion module and the output module. the output † https://cloud.baidu.com/ module can be the action module or the response module. ## conclusion in this paper, we proposed the task of gui-based task-oriented dialogue system, which replaces the traditional tod-specific api calls with gui operations on real apps. the advantage is that intelligent agents can perform tasks without the need of backend tod-specific apis and it doesn't rely on a domain-specific schema, which means it can be applied to a new domain easily. we collect meta-gui, a dataset with dialogues and gui traces to serve as a benchmark. our model shows promising results on the dataset, and we hope this work could stimulate more advanced methods on gui-tod. in the future, we will explore how to better incorporate gui traces into our model and build the gui semantics based on interactions.