Situated and Interactive Multimodal Conversations (2020)

Abstract: Next generation virtual assistants are envisioned to handle multimodal inputs (e.g., vision, memories of previous interactions, and the user’s utterances), and perform multimodal actions (, displaying a route while generating the system’s utterance). We introduce Situated Interactive MultiModal Conversations (SIMMC) as a new direction aimed at training agents that take multimodal actions grounded in a co-evolving multimodal input context in addition to the dialog history. We provide two SIMMC datasets totalling ~13K human-human dialogs (~169K utterances) collected using a multimodal Wizard-of-Oz (WoZ) setup, on two shopping domains: (a) furniture – grounded in a shared virtual environment; and (b) fashion – grounded in an evolving set of images. Datasets include multimodal context of the items appearing in each scene, and contextual NLU, NLG and coreference annotations using a novel and unified framework of SIMMC conversational acts for both user and assistant utterances. Finally, we present several tasks within SIMMC as objective evaluation protocols, such as structural API prediction, response generation, and dialog state tracking. We benchmark a collection of existing models on these SIMMC tasks as strong baselines, and demonstrate rich multimodal conversational interactions. Our data, annotations, and models will be made publicly available.

## introduction as virtual digital assistants become increasingly ubiquitous, they are expected to be embedded in the day-to-day life of users the same way a human assistant would. we thus envision that the next generation of virtual assistants will be able to process multimodal inputs and provide multimodal outputs beyond the traditional nlp stack. to this end, we present situated interactive multimodal conversations (simmc) tasks and datasets as a starting point in this new research direction. specifically, simmc focuses on task-oriented dialogs that encompass a situated multimodal context, where situated implies that the user and assistant are continually co-observing the same context, and that context can be updated on each turn. we provide two new simmc datasets in the domain of interactive shopping, collected using the simmc platform @xcite : (1) furniture and (2) fashion. moreover, we provide fine-grained annotations to allow for both end-to-end and component-level modelling. the annotation includes natural language understanding (nlu), multimodal-coreference, multimodal state tracking, assistant actions, natural language generation (nlg), and item appearance logs. fig. 1 illustrates an exemplary dialog from our simmc-furniture dataset, where a user is interacting with an assistant with the goal of browsing for furniture. in our setting, the assistant can update the coobserved environment to create a new multimodal context based on the preceding dialog with the user, e.g., visually presenting recommended chairs in a virtual reality (vr) environment, or responding to the request "i like the brown one. show me the back of it." by executing the actions of focusing on, and rotating the indicated item. these actions change the shared multimodal context, which grounds the next figure 1 : illustration of a simmc dialog: a user and an assistant interact in a co-observed multimodal environment for a shopping scenario. the dialog is grounded in an evolving multimodal context. the ground-truth of which items (e.g., prefabs) appear is known for each view. ## novelty & related work tab. 1 presents main distinctions of simmc compared to the the existing multimodal dialog datasets. multimodal datasets for co-observed, real-world assistant. with the ultimate goal of laying the foundations for the real-world assistant scenarios, we assume a co-observed multimodal context between a user and an assistant. this shifts the primary focus onto the core problem of grounding conversations in the co-observed multimodal context. in contrast, the existing literature @xcite @xcite , drawing motivation from the visual question answering @xcite , posits the roles of a primary and secondary observer, i.e., questioner and answerer, who do not co-observe the same multimodal context. in addition, we study scenarios in which the situated multimodal context is dynamically updated, reflecting the agent actions. in our settings, agent actions can be enacted on both the object-levelchanging the view of a specific object within a scene, and the scene-level -introducing a new scene or an image. while the dialog-based image retrieval tasks @xcite and the visual navigation tasks @xcite do comprise context updates, they are limited to the introduction of new visual scenes, e.g., new images or locations. focus on task-oriented dialogs. we frame the problem as a task-oriented, multimodal dialog system, with the aim of extending the capabilities of digital assistants to real-world multimodal settings. while one focus area of the dialog community is on task-oriented dialog, which has practical applicability consumer-facing virtual assistants @xcite @xcite @xcite , this form of dialog is often neglected in many existing multimodal 'dialog' datasets (both in terms of the task design and the annotations), where the primary focus lies in visual grounding of language. our work aims to bring important challenges actively studied in the dialog community to the multimodal setting. specifically, we study the multimodal extension of the traditional dialog state tracking (dst) and the assistant api prediction tasks, which have been the key focus of dialog literature @xcite @xcite . compared to the conventional task-oriented dialog datasets (e.g., multiwoz @xcite ), the agent actions in simmc span across a diverse multimodal action space (e.g., rotate, search, add to cart). our study thus shifts the focus of the visual dialog research from the tokenlevel grounding of visual scenes to the task-level understanding of dialogs given multimodal context. semantic annotations for multimodal dialogs. finally, we present a novel flexible schema for semantic annotations that we developed specifically for the natural multimodal conversations. the proposed simmc annotation schema allows for a more systematic and structural approach for visual grounding of conversations, which is essential for solving this challenging problem in the real-world scenarios. to the best of our knowledge, our dataset is the first among the related multimodal dialog corpora to provide fine-grained semantic annotations. ## simmc datasets for simmc datasets, we focused on the shopping domain as it often induces rich multimodal interactions around browsing visually grounded items. as shown in fig. 1 , the setup consists of two human workers, a user and an assistant, conversing around a shopping scenario. the goal of the user is to interactively browse through an inventory of items while that of the assistant is to facilitate this conversation. in addition to having an interactive dialog, the assistant manipulates the co-observed environment to show off items from the shopping inventory. a conversational assistant model for the simmc datasets would need to (i) understand the user's utterance using both the dialog history and the state of the environment -the latter provided as multimodal context, and (ii) produce a multimodal response to the user utterance, including updates to the co-observed environment to convey meaningful information as part of the user's shopping experience. we provide two simmc datasets with slightly different setups and modalities. see tab. 2 for overall statistics. the simmc-furniture (vr) dataset captures a scenario where a user is interacting with an assistant whilst browsing for furniture, e.g., couch, or side table. grounded in a vr environment (unity technologies, 2019) the assistant can manipulate items in the scene while engaging in conversation. we seed the conversation by presenting the user with either a high-level directive such as 'shop for a table' or an image of a furniture item to shop for. the user is then connected randomly with a human assistant. the assistant can filter the catalog by attributes such as furniture category, price, color, and material, navigate through the filtered results and share their view with the user. as part of the dialog, the user can request to look closer at one of the options, or see other options. in response, the assistant can either zoom into an item, present an alternate view by rotating it, or look at the catalog description to answer further questions. to enable this, the environment is designed to transition between two states: (a) carousel, which displays three filtered furniture items (top view, fig. 1 ); and (b) focused, which provides a zoomed in view of one item from the carousel view (bottom view, fig. 1 ). the conversation continues for 6-12 turns until the user considers that they have reached a successful outcome. tab. 10 shows example dialogs. the simmc-fashion (image) dataset represents user interactions with an assistant to obtain recommendations for clothing items, e.g., jacket, dress. conversations are grounded in real-world images that simulate a shopping scene from a user's point-of-view (pov). at the start of each dialog the user is presented with a randomly selected 'seed' image from the catalog to emulate (visually) that they are in the middle of shopping, as well as a sequence of synthetic memories of 'previously viewed items'. in addition to the user's context, the assistant has access to a broader catalog that allows for information lookup and item recommendation. we ask the user to browse and explore options by asking the assistant for recommendations based on the shared attributes, preferences, as referred from visual scenes, memories, and assistant-recommend items. the conversation continues for 6-10 turns until the user is assumed to be given a successful recommendation. please refer to tab. 11 for example dialogs. for both datasets, the ground-truth of which items appear in each view is logged and included in the multimodal context. this allows the problem of computer vision to be sidestepped and focus on semantically combining the modalities. the datasets were collected through the simmc platform @xcite , an extension to parlai @xcite for multimodal conversational data collection and system evaluation. note that even though we focus on english in this work, our data collection framework is language-agnostic and can be easily extended to other languages. ## simmc dialog annotations building a task-oriented multimodal conversational model introduces many new challenges, as it requires both action and item-level understanding of multimodal interactions. while most of the previous multimodal datasets provide surface-level annotations (e.g., utterance to multimodal action pairs), we believe it is critical to provide the semantic-level fine-grained annotations that ground the visual context, allowing for a more systematic and structural study for visual grounding of conversations. towards this end, we develop a novel simmc ontology that captures the detailed multimodal interactions within dialog flows. note that these dialog annotations are collected after the dialog data collection stage, with the help of professional linguists. in this section, we describe the proposed simmc ontology and the hierarchical labeling language centered around objects (sec. 4.1 and 4.2), and the multimodal coreference schema that links the annotated language with the co-observed multimodal context (sec. 4.3). ## simmc tasks & metrics we define several offline evaluation tasks within the simmc framework to train conversational models on these new datasets using the fine-grained annotations that are provided. we first provide the general offline evaluation framework for defining simmc tasks (sec. 5.1), and then present three major tasks that we focus on in this paper. these are primarily aimed at replicating human-assistant actions in order to enable rich and interactive shopping scenarios (sec. 5.2). ## modeling for simmc tasks we now propose several models building on top of prior work and train them on the tasks formulated in sec. 5 to benchmark the simmc dataset. we define two classes of models for the simmc tasks: (1) assistant models, which aims at mimicking the assistant actions and responses (task 1 & 2), and (2) user belief tracking model (task 3) that output semantic parses of user utterances, agnostic of future assistant actions. our principal assistant model architecture is illustrated in fig. 3 , which is composed of four main components: utterance and history encoder, multimodal fusion, action predictor, and response generator. on the other hand, our user belief model builds upon the state-of-the-art dst models and extend them to accommodate for multimodal input. inspired by (hosseini-asl et al., 2020), we adapt one such model and finetune a pretrained gpt-2 language model @xcite to both action prediction and belief tracking. utterance & history encoder. the utterance and history encoder takes as input the user utterance at the current round u t and the dialog history so far h t , to produce the utterance encoding u t and history encoding h t to capture the respective textual semantics. inspired from prior work, we consider several utterance and history encoders, whose functional forms are outlined in tab. 3. we embed each token in the input sequences (u t or h t ) through learned word embeddings of size d w , which are further fed into the encoders. these output u @xmath1, where d h is the embedding size, n u is length of the user utterance. (a) history-agnostic encoder (hae) ignores the dialog context h t to only encode the user utterance through an lstm @xcite for the downstream components. (b) hierarchical recurrent encoder (hre) @xcite ) models dialogs at two hierarchical recurrence levels of utterance and turn. the utterance encoder lstm operates at the former, while a history lstm that consumes the hidden states of utterance encoder lstm from all the previous rounds ([u i ] t-1 @xmath2. (c) memory network (mn) encoder @xcite treats dialog history h t as a collection of memory units comprising user and assistant utterance pairs concatenated together, and uses the current utterance encoding u t to selectively attend to these units to produce the utterance-conditioned history encoding h t . (d) transformer-based history-agnostic encoder (t-hae) is a variant of hae with the lstms replaced with transformer units @xcite that achieved state-of-the-art results in language modeling @xcite . multimodal fusion. this component fuses semantic information from the text (u t and h t ) and the multimodal context m t (described in sec. 7), to create the fused context tensor ũt ∈ r n u ×2d h , which is double the size of u t in the last dimension. in our setup, the multimodal context is modelled as a tensor of size m @xmath3, where n m is the number of multimodal units for the current round t and d m is the multimodal embedding size. note that all of our models have the same architecture to fuse multimodal information. at a high level, we first embed m t to match its size to d h using a linear layer followed by a non-linearity (relu) (eq. 1), then use the utterance encoding u t to attend to the multimodal units (eq. 2), and finally fuse the attended multimodal information with ũt = [u t ; m t ]. more concretely, action predictor. using the fused context ũt , the action predictor predicts the appropriate action (api) ât and the corresponding api arguments to be taken by the assistant. the former is a multiclass classification that chooses from a set of actions (apis) while the latter is a multi-way classification modelled as a set of binary classifiers one for each attribute like category, color, price, etc. for a list of apis and their arguments supported in our work, see tab. 8 in appendix d. first, the tensor ũt is transformed into a vector q @xmath4. 4). next, we learn a classifier (mlp) that takes in q t to predict the distribution over the possible apis (eq. 5). in addition, we also learn several binary classifiers (mlp) one each for the corresponding api arguments. having predicted the structured api calls, we execute them and encode the output as action context c @xmath5, where n a is the number of context units and d a is action context embedding size. the dataset-dependent specifics about the api call output encoding c t are in sec. 7. finally, c t and ũt feed into the last component to generate the assistant response. as the training objective, we minimize the cross entropy loss l a for both the action and action attributes. response generator. as the last component, the response generator (decoder) generates the assistant response ât . in our work, we model it as a language model conditioned on both c t and ũt . the former ensures that the response is influenced by the api call output while the latter maintains semantic relevance to the user utterance. for example, the response to 'show me black couches less than @xmath0500'. for models that use lstm for user and history encoders, the response decoder is also an lstm with attention over fused context ũt and action api output c t at every decoding time step, similar to @xcite . similarly, we use a transformer-based decoder for the other models to ensure consistent underlying architecture (either lstm or transformer). like any conditional language model, we decode individual tokens at each time step to generate ât , and minimize the negative loglikelihood l a of the human response under the model during training. ## experiments & results dataset splits and baselines. our models are learned on randomly sampled train (60%), model hyperparameters chosen via early stopping using performance on dev (10%), and evaluation numbers reported on the unseen testdev (15%). in addition to the models described in sec. 6, we consider two simple baselines that use tf-idf features for utterance and history encoders for action prediction, and lstm-based language model (lstm) trained solely on assistant responses, and compare against them. ## conclusion in this work, we presented situated interactive multi-modal conversations (simmc), an important new direction towards building next generation virtual assistants with evolving multimodal inputs. in particular, we collected two new datasets using the simmc platform, and provided the contextual nlu and coreference annotations on these datasets, creating a new simmc task for the community to study. we established several strong baselines for some of the tasks enabled by the datasets, showcasing various uses of the datasets in real-world applications. the fine-grained annotations we collected open the door for studying several different tasks in addition to the ones highlighted in this work, which we leave as future work for the community to tackle.