Overview of Situated and Interactive Multimodal Conversations ( SIMMC ) 2.1 Track at DSTC 11 (2023)

Abstract: With ever increasing interest in task-oriented dialog systems, the recent work on Situated and Interactive Multimodal Conversations (SIMMC 2.0) aims to develop personal assistants that interact with users, grounded in an immersive and co-observed setting of photo-realistic scenes. The dataset contains 11k task-oriented dialogs set in an interactive shopping scenario, spanning more than 117k utterances. In order to push research towards this next generation virtual assistants, the SIMMC 2.1 challenge was conducted at the Eleventh Dialog System Technology Challenge (DSTC) which had entries from across the world competing to achieve the state-of-the-art performance in the SIMMC 2.1 task. In this report, we present and compare 13 SIMMC 2.1 model entries from 5 trams across the world to understand the current progress made across the last three years (starting with SIMMC 1.0 and 2.0 challenges) for multimodal task-oriented dialog systems. We hope that our analysis throws light on components that showed promise in addition to identifying the gaps for future research towards this grand goal of an immersive multimodal conversational agent.

## motivation the situated and interactive multimodal conversaitional ai (simmc) challenges @xcite , held as part of dstc9 @xcite and dstc10, pioneered the work for building the real-world assistant agents that can understand multimodal inputs (vision & conversations) and handle user requests. throughout the two editions of the challenge, we provided two new benchmark datasets (simmc 1.0 and 2.0) for studying multimodal conversations with situated user context in the form of a co-observed image or virtual reality (vr) environment. specifically, the simmc 2.0 dataset provided the assistant↔user task-oriented dialogs grounded on diverse photo-realistic vr renders of (synthetic) commercial stores with various referent objects, serving as a proxy for complex realworld scenarios. the earlier simmc challenge at dstc10 saw a total of 16 model entries from participants around the world, establishing a new set of state-of-the-art baselines for the multimodal task-oriented dialog systems. while the new sota models have drastically improved the performance on the previous benchmark tasks, several challenges remain in building the production-ready agent. one such challenge is the visual disambiguation which is often encountered in real-world multimodal conversations. for instance, the user ambiguously uses 'these two trousers' in fig. 1 , which cannot be deterministically resolved. thus, an assistant system needs to reason about this, identify the possible ambiguous candidates, and then ask a follow-up question to disambiguate. this is important to avoid a wrong resolution of such references leading to subsequent turns with false premises. another advantage is that the assistant could insert implicit confirmation -"which of the two red jackets are you referring to -the one on the left or the one closer to you?", as opposed to a generic response like: "which one are you referring to?". to this end, we conducted the third edition of the simmc challenge (simmc 2.1) for the community to tackle and continue the effort towards building a successful multimodal assistant agent. in this edition of the challenge, we specifically focused on the key challenge of fine-grained visual disambiguation, which adds an important skill to assistant agents studied in the previous simmc challenge. to accommodate for this challenge, we provided the improved version of the of the dataset, simmc 2.1, where we augment the simmc 2.0 dataset with additional annotations (i.e. identification of all possible referent candidates given ambiguous mentions) and corresponding re-paraphrases to support the study and modeling of visual disambiguation (simmc 2.1). this new version of the dataset poses several interesting challenges such multimodal dialog state tracking given ambiguity, coreference resolutions ("directly behind it", "grey jacket to the right of the one i mentioned"), and disambiguation strategies ("how much is that shirt" → "which shirt are you referring to?"). ## the third simmc challenge we now detail the new version of the multimodal conversational dataset (simmc 2.1) (sec. 2.2) and the four main sub-tasks (tab. 1, sec. 2.3). ## baselines there are two baselines, adopted from @xcite : specifically for the task of ambiguous candidate identification (subtask 1), we extract the feature representation of the last token as the textual feature and compute similarity with the visual features extracted for each item in a given scene. visual features for the items are extracted similar to the above mtn network. ## conclusion the goal of the simmc 2.1 challenge in dstc11 is to motivate and inspire the research community to work on the problem of creating dialog agents that can handle multiple modes of communication and handle tasks in a specific context. these agents have many practical uses and present a range of challenges related to multimodal communication. the simmc 2.1 challenge specifically saw a number of submissions exploring different transformer architectures, highlighting the trade-offs in performance. we hope that the insights gained from this challenge will shed light on the challenges of multimodal dialogs and inspire further research in this area. what next? we identify possible future directions for the simmc challenge to continue advancing in this area of research.