MUG : Interactive Multimodal Grounding on User Interfaces (2024)

Abstract: We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen. Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command. Yet, in a realistic scenario, a user command can be ambiguous when the target action is inherently difficult to articulate in natural language. MUG allows multiple rounds of interactions such that upon seeing the agent responses, the user can give further commands for the agent to refine or even correct its actions. Such interaction is critical for improving grounding performances in real-world use cases. To investigate the problem, we create a new dataset that consists of 77,820 sequences of human user-agent interaction on mobile interfaces in which 20% involves multiple rounds of interactions. To establish benchmark, we experiment with a range of modeling variants and evaluation strategies, including both offline and online evaluation—the online strategy consists of both human evaluation and automatic with simulators. Our experiments show that iterative interaction significantly improves the absolute task completion by 18% over the entire test set and 31% over the challenging split. Our results lay the foundation for further investigation of the problem.

## introduction natural language understanding on graphical user interfaces (guis) is crucial for realizing human-computer interaction and assisting scenarios that have accessibility difficulties @xcite . specifically, interpreting user commands into executable actions has drawn increasing interests as it manifests rich research problems including multimodal modeling and natural language grounding (e.g., @xcite @xcite @xcite . prior works often consider ui grounding in a single-pass fashion where the model predicts actions with a given instruction without looking backward to refine prediction. however, in a realistic scenario, user instructions can be ambiguous or imprecise when the target action is difficult or inconvenient to articulate. reasoning in such cases is inherently iterative. therefore, it is important and beneficial to incorporate interaction for resilient grounding @xcite . in this paper, we investigate interactive grounding on guis, which aligns multimodal input to actionable objects of a screen. we focus on singlescreen interaction which is the building block of ui reasoning. specifically, we introduce the mug (multi-turn ui grounding) task in which the user iteratively guides the agent to select a desired ui object (see fig. 1 ). with a given ui and a target object, the user instructs the agent via natural language, ranging from casual intent to more descriptive commands. the agent infers which ui object is intended by the user and and highlights it. if the agent is correct, the user can confirm the selection and the grounding is completed. otherwise, the user issues further guidance, e.g., "click the one below", to the agent to refine its selection. we collecte the mug dataset from live interaction sessions between pairs of human annotatorsone acts as the user and the other as the agent. our dataset has 77, 820 examples, each records the transaction history in a session. specially, 20% of the dataset are challenging ones as their human commands need multiple rounds to ground, even for human agents. to establish the benchmark, we experiment with a range of variants to model the dynamics between the two roles. while the main goal of the task is to develop agent models for grounding, we also develop the user models for online instruction simulation. we build our models upon a transformer-based encoder-decoder architecture @xcite , and experiment with in summary, our key contributions 1 are: ## background multi-modal modeling has a long history of research (e.g., @xcite @xcite @xcite . one important area focuses on grounding objects in images where the natural language is used as an additional input @xcite @xcite @xcite . interactive multimodal grounding prior works have formulated grounding as a multi-step reasoning task, e.g., navigation via multiple steps of grounding (e.g., @xcite . our work differs by focusing on agent's ability to self-correct in synchronized turns of interaction on a ui screen. it is also conceptually linked to repeated reference game @xcite , except we use a different form of communication (language-action) instead of dialogue (language-language). our task leverages iteratively refined instructions on atomic action instead of the increased instruction utility over multi-step actions @xcite . we model both the user and the agent, and let them communicate online. this is different from single-sided modelings @xcite . our observation that interaction improves grounding is also in line with dialoguebased works (e.g., @xcite . ## task formulations as a grounding task, mug involves two participants: a user and an agent. our formulation includes both roles to provide a holistic view of interactive grounding. the user's goal is to instruct, via natural language, the agent to select the desired object g on the ui screen s. the unique aspect of mug is that it allows the user to guide the agent iteratively to identify the target action by issuing a series of commands, each in response to the agent's prior inference. we separate such user-agent interaction into turns. at turn t, the interaction consists of: c t : user command, a t : agent action. ## dataset creation as there is no available dataset for model training and evaluation, we developed an interactive labeling interface to collect data for mug. our data collection involves two human annotaters to play the roles of the user and the agent respectively in a live session. the user and the agent have two separate views, running on different machines (appx. a). both views share the same ui screen and a message box showing instruction history. our task embodies the eyes-on, hands-free situation for mobile interaction where the user is required to only use language for the task, and the machine responds its prediction by highlighting. the user can commit the action if the prediction is confirmed. in a session, only the user can see the target; and the message box is read-only to the agent so no language-base dialogue would happen. ## grounding models we aim to have a general architecture for the ui domain and explore its variants to model multiturn interaction. our agent model is based on a transformer encoder-decoder network, inspired by @xcite . specifically, we extend the architecture to handle interaction history as input in the decoder. ## analysis tab. 8 shows how model predictions are affected by corrective instructions generated by heuristics or the neural instruction model. our agent models make repeatedly the same incorrect selection, irrespective of the corrective instruction. even considering the entire test set, there are @xmath0. we broadly attribute this observation to the difficulty of the task as well as the challenge in multimodal modeling. tab. 9 compares our heuristics-based online evaluation against immediate alternatives. the large and consistent performance gaps suggest our agent models follow the hints in corrective instructions instead of random-guessing. for brevity, we used the multi model to demonstrate. other multiturn models performed in a similar pattern (e.g., 15∼20% better f1 4 with heuristics). in appx. i, we demonstrate predictions from the imitation model with successfully solved examples as well as failed ones. ## conclusions in this paper, we presented mug, a novel and challenging task for multimodal grounding on ui. mug requires a grounding agent being able to correct its own prediction, and allows a user to guide the agent via natural language instructions. for the task, we contribute a new dataset, investigate modeling options for the agent, and propose evaluation strategies along with two user models for automatic online testing. we found that interaction greatly improves grounding accuracy in the ui domain. our experiments and analyses also suggest large room for grounding performances, even on a seemingly easy single screen task, which calls for future investigation. our work also contributes to the general effort of multimodal language understanding and its robustness by enabling synchronized multi-turn interactivity.