SIMMC - VR : A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams (2023)

Abstract: Building an AI assistant that can seamlessly converse and instruct humans, in a user-centric situated scenario, requires several essential abilities:(1) spatial and temporal understanding of the situated and real-time user scenes,(2) capability of grounding the actively perceived visuals of users to conversation contexts,and (3) conversational reasoning over past utterances to perform just-in-time assistance. However, we currently lack a large-scale benchmark that captures user–assistant interactions with all of the aforementioned features. To this end, we propose SIMMC-VR, an extension of the SIMMC-2.0 dataset, to a video-grounded task-oriented dialog dataset that captures real-world AI-assisted user scenarios in VR.We propose a novel data collection paradigm that involves(1) generating object-centric multimodal dialog flows with egocentric visual streams and visually-grounded templates,and (2) manually paraphrasing the simulated dialogs for naturalness and diversity while preserving multimodal dependencies. To measure meaningful progress in the field, we propose four tasks to address the new challenges in SIMMC-VR, which require complex spatial-temporal dialog reasoning in active egocentric scenes. We benchmark the proposed tasks with strong multimodal models, and highlight the key capabilities that current models lack for future research directions.

## introduction with the growing popularity of smart glasses, studies on visually grounded conversational agents have gained significant interest. for instance, simmc 2.0 @xcite introduces an imagegrounded, task-oriented dialog (tod) dataset where an assistant agent co-observes the user's egocentric viewpoint to aid with user requests. many follow-up works @xcite @xcite focus on challenges around dialog-image grounding, such as figure 1 : simmc-vr is a situated interactive multi-modal conversation dataset that features task-oriented user↔assistant dialogs streamed immersively in a virtualreality (vr) environment. the dataset is created on programmed realistic shopping scenarios and actively-rendered photorealistic user visual observations, which brings new challenges for complex spatial-temporal reasoning on the multimodal interactions (visual cues and grounded-dialogs). however, several technical gaps still remain in applying prior work to build a real-world, situated multimodal assistant (figure 1 ). for instance, a typical multimodal user-assistant scenario (with a video capturing capability) would include (1) spatial and temporal language references as grounding contexts ('the shirt i saw earlier when i entered the store'), (2) actively perceived egocentric motions as part of conversation contexts ("no -turn around the other way"), (3) references to conversational memories from past sessions ('the one i bought earlier', the 'black coat' in figure 1 being retroactively mentioned by both the assistant and the user), etc. while these scenarios are perceived as the expected capabilities of a next-generation multimodal assistant, our survey of datasets (sec. 5) highlights that due to the static and constrained nature of the datasets' grounding context, they lack sufficiently complex interactions. to this end, we present simmc-vr, a videogrounded task-oriented dialog dataset comprising 4k user↔assistant task-oriented dialogs (95.3k utterances) grounded on diverse photorealistic vr video streams (4.8m frames). for data collection, we propose a novel two-stage approach with: (1) a multimodal interaction simulator that generates egocentric vr streams grounded on object-centric multimodal dialog flows, and (2) a manual paraphrasing step for naturalness and diversity while preserving multimodal dependencies between visual scenes and their grounding language. our pipeline allows for flexible and cost-effective data collection, easily extendable to simulate any other domains given the availability of 3d virtual assets. to measure progress towards real-world applicability, we propose four simmc-vr tasks that address new challenges in complex spatio-temporal dialog reasoning. we then extend state-of-the-art multimodal models to the simmc-vr tasks and discuss the limitations of current models. our contributions are as follows: (1) we present simmc-vr, a video-grounded task-oriented dialog dataset (95k utterances over 4.8m frames) targeted towards real-world applications for an assistant on smart glasses. (2) we propose the tasks with complex spatio-temporal conversational dependencies, and benchmark them by extending the state-ofthe-art multimodal models. (3) our data collection platform allows creation of a similar dataset in any target domains. ## simmc-vr dataset simmc-vr is actively multimodal, where each data instance is a video from a user's egocentric viewpoint recording all interactions within a virtual shopping environment, densely paired with dialog utterances and essential attributes. each taskoriented dialog mimics real-world shopping scenarios where the assistant's goal is to help the user make purchases and navigate through the environment. in each instance, the user walks around a virtual shop while the assistant provides product meta-agenda goal generator user simulator asst. simulator information or recommendations; as well as help the user locate and navigate to products of interest. ## simmc-vr task formulation the simmc-vr is created to help ai models cope with realistic shopping scenarios and assist human users in real-world applications in ar/vr. to investigate the (multimodal) conversational and assistive abilities of current ai systems in this immersive and situated environment, we propose four main bechmarking tasks leveraging the created dataset. several tasks inherit from simmc 2.0 with additional challenges brought by the nature of active user scenes and expanded dataset annotations. ## modeling & experimental analysis in this section, we introduce the investigated baseline models to perform a preliminary benchmarking of the proposed dataset, where we hope to inspire more sophisticated and tailored modeling efforts from the community for future research. dataset split. for the empirical modeling analysis and performance benchmarking, we randomly split the dataset into 3 sets: train (70%), dev (5%), and test (25%) sets, while ensuring both domains (fashion and furniture) have the same split distributions. baselines. to benchmark the dataset, we adopt: (a) mm-dst model is a 12-layered multi-task gpt-2 model @xcite trained with joint supervision signals from mm-coref, mm-dst, and response generation tasks, inspired by causal language modeling approach to dialog systems @xcite . the inputs to the model include both the dialog context (utterances) and the multimodal contexts flattened as structurally formatted text strings, where the outputs are the predicted dst labels. this baseline has two versions: one uses the ground-truth multimodal contexts provided from the scene generator (hence a soft oracle) to simulate the outputs from a robust object detector or from a controlled vr environment, whereas the other has to infer visual descriptors from raw videos, simulating real-world scenarios. ## related work the proposed work addresses unique requirements for a task-oriented assistant on smart glasses, making it a first-of-its-kind -while complementing other related works within multimodal nlp. simmc @xcite is a class of research areas that the proposed work builds upon, which addresses using virtual environments to simulate a co-observing multimodal dialog agent. moving away from the sanitized and static scenes that they concern for the limited use cases, simmc-vr introduces several additional challenges as summarized in section 2.3. several models @xcite @xcite are proposed for the simmc benchmark tasksprimarily focusing on grounding dialogs on visual objects from a single image. taking inspirations from these works, we extend the models to accommodate temporal dependencies within frames. multimodal dialog datasets. many of the existing literature in multimodal dialogs @xcite @xcite @xcite typically assume asymmetric visual information between two observers, i.e. questioner and answerer, where conversational goals are limited to reducing information asymmetry (similar to vqa). in contrast, we study task-oriented dialog scenarios -an assistant co-observes the same scene as a user does, thus focusing on serving user requests to achieve functional goals (e.g. giving recommendations). the embodied ai dialog systems @xcite , on the other hand, study the scenarios where a human participant teaches an ai agent a set of skills or gives navigational directions -hence posing an opposite role to an ai agent. while it is an important area to study, its distribution of utterance patterns is completely different and therefore not applicable for our target domain -building a situated ai assistant. egocentric video datasets. with the popularity of wearable devices, several datasets @xcite @xcite are released to study the unique properties of egocentric videos. our work also features similar visual properties, while adding conversational layers that showcase an assistant use case of such videos. task-oriented dialog systems @xcite @xcite have long been studied to support various assistant scenarios (e.g. booking hotels). our work takes its roots in this line of work -focusing on predicting user belief states and dialog acts to achieve functional goals -and extends it to a unique multimodal setting. a popular thread in the task-oriented dialog system modeling is to fine-tune end-to-end causal llms @xcite @xcite @xcite . we extend this line of work and propose a multimodal extension to account for visual inputs. ## conclusions we present simmc-vr, a situated and interactive dialog dataset that features immersive vr streams as multimodal contexts, simulating realistic shopping scenarios along with user-assistant dialog interactions. the dataset consists of 4k user-egocentric videos paired with densely annotated dialog utterances. we build a novel meta-agenda generator for automatically synthesizing rich interactive dialogs grounded on active and diverse visual scenes, paraphrased manually for more natural speech. we propose four sub-tasks on simmc-vr which aims at inspiring future dialogue modeling endeavors on high-fidelity egocentric (user pov) environments; where the baseline performance highlights many challenges the dataset brings forth towards actualizing the real-world-ready vr/ar assistant. with rich annotations it provides, simmc-vr can as well expand beyond the proposed tasks to spur relevant future research, which includes (but not limited to): (1) augmented with speech-like spoken utterance interventions to enrich the naturalness of the dialogues, and (2) environments and room layouts beyond ones used under the scope of this paper.