SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations (2021)

Abstract: Next generation task-oriented dialog systems need to understand conversational contexts with their perceived surroundings, to effectively help users in the real-world multimodal environment. Existing task-oriented dialog datasets aimed towards virtual assistance fall short and do not situate the dialog in the user’s multimodal context. To overcome, we present a new dataset for Situated and Interactive Multimodal Conversations, SIMMC 2.0, which includes 11K task-oriented user<->assistant dialogs (117K utterances) in the shopping domain, grounded in immersive and photo-realistic scenes. The dialogs are collection using a two-phase pipeline: (1) A novel multimodal dialog simulator generates simulated dialog flows, with an emphasis on diversity and richness of interactions, (2) Manual paraphrasing of generating utterances to draw from natural language distribution. We provide an in-depth analysis of the collected dataset, and describe in detail the four main benchmark tasks we propose for SIMMC 2.0. Our baseline model, powered by the state-of-the-art language model, shows promising results, and highlights new challenges and directions for the community to study.

## introduction the situated and interactive multimodal conversational ai (simmc) challenge @xcite , held as part of dstc9 @xcite , aimed to lay the foundations for the real-world assistant agents that can handle multimodal inputs, and perform multimodal actions. specifically, it provided the simmc datasets as new benchmarks for studying task-oriented dialogs that encompass a situated multimodal user context in the form of a co-observed image or virtual reality (vr) environment. since the introduction of the dataset, a ⇤ joint first authors 1 code & data are made publicly available: https:// github.com/facebookresearch/simmc2 number of follow-up work @xcite @xcite @xcite have established a new set of state-of-the-art baselines for the multimodal taskoriented dialog systems on simmc. though simmc serves as a step towards building multimodal virtual agents, the dataset falls short (understandably so) in the complexity of the considered multimodal contexts. in particular, the coobserved image or vr environment is simplistic and far from realistic user situations. to bridge this gap, we take inspiration from the first simmc challenge @xcite and propose a new multimodal dialog dataset (simmc 2.0) for the community to tackle and continue the effort towards building a successful multimodal assistant agent. specifically, simmc 2.0 is designed to include a closer-to-real-world context for a fashion or furniture shopping scenario, moving away from the sanitized contexts present in the original simmc datasets. to this end, we propose a vr scene generator that allows for controlling and capturing diverse multimodal contexts with ground-truth scene graph information, while serving as a close proxy for real-world scenarios. we then collect 11k assistant$user task-oriented dialogs (117k utterances) grounded on diverse photo-realistic vr renders of commercial stores (1.5k different scenes). the incorporation of the complex and cluttered multimodal contexts introduces several interesting challenges, such as understanding visual and dialog coreferences ('the one directly behind it', 'the one i mentioned'), tracking dialog states along with multimodal objects, etc. in addition, the use of photorealistic scenes surfaces practical limitations of cv models that need to be addressed, such as the detection of partially observed or obstructed referent objects, the visual texture recognition, etc. to this end, we propose four main benchmark tasks that are essential in building a multimodal task assistant: multimodal disambiguation, multimodal coreference resolution (mm-coref), multimodal dialog state tracking (mm-dst), and response generation. we then provide a baseline model trained for these tasks, and highlight the key challenges and future research directions. ## related work problem setup: the simmc 2.0 dataset addresses the conversational scenarios where the virtual assistant shares a co-observed scene with a user in addition to the traditional communication that takes place in the form of natural language. specifically, we choose the shopping experience as the domain for this study, as it often induces rich multimodal interactions around browsing visually grounded items. we assume that the assistant agent has ground-truth meta information of every object in the scene, while users only observe those objects through the visual modality to describe and compose a request. in addition, we allow users to physically navigate within each scene, which we simulate as multiple viewpoints updated at different time steps throughout each dialog. thus, models for simmc 2.0 would need to understand the user utterance using both the dialog history and the state of the environment as multimodal context. multimodal dialog datasets: note that our problem setup for co-observing assistant scenarios allows for more natural multimodal coreferences to be used as part of user-assistant conversations. the existing literature in multimodal dialogs @xcite @xcite @xcite often posits the roles of a primary and secondary observer, i.e. questioner and answerer similar to the visual question answering @xcite tasks, hence showing a different distribution of language. task-oriented dialog systems: many datasets have been developed in the past to support various assistant scenarios (e.g. booking hotels, reserving hotels) @xcite @xcite , defining many challenges in handling user requests under the unimodal dialog setting. our setup extends many of these challenges studied in the previous literature on task-oriented dialog systems (e.g. dst, slot carryovers) to the unique multimodal settings. the most recent thread in building a taskoriented dialog system is to fine-tune an end-toend system on a large pre-trained causal language model, which achieves the state-of-the-art performance in many metrics @xcite @xcite @xcite . we follow this line of work and provide a baseline which extends it to accommodate for the multimodal input. ## task formulation the aim of the simmc 2.0 dataset is to emulate futuristic, real-world shopping scenarios where hu-mans converse with a dialog agent in natural language grounded in a situated multimodal context. as a step towards this intelligent conversational agent, we leverage the dialogs and annotations in our dataset and propose four benchmark tasks (summarized in tab. 4) along with evaluation metrics. these tasks capture several multimodal conversational reasoning challenges, as elaborated next. ## modeling & empirical analysis in this section, we perform preliminary empirical analysis and train baselines. we leave more detailed modeling work for the future.