An Analysis of State-of-the-Art Models for Situated Interactive M ulti M odal Conversations ( SIMMC ) (2021)

Abstract: There is a growing interest in virtual assistants with multimodal capabilities, e.g., inferring the context of a conversation through scene understanding. The recently released situated and interactive multimodal conversations (SIMMC) dataset addresses this trend by enabling research to create virtual assistants, which are capable of taking into account the scene that user sees when conversing with the user and also interacting with items in the scene. The SIMMC dataset is novel in that it contains fully annotated user-assistant, task-orientated dialogs where the user and an assistant co-observe the same visual elements and the latter can take actions to update the scene. The SIMMC challenge, held as part of theNinth Dialog System Technology Challenge(DSTC9), propelled the development of various models which together set a new state-of-the-art on the SIMMC dataset. In this work, we compare and analyze these models to identify‘what worked?’, and the remaining gaps;‘whatnext?’. Our analysis shows that even though pretrained language models adapted to this set-ting show great promise, there are indications that multimodal context isn’t fully utilised, and there is a need for better and scalable knowledge base integration. We hope this first-of-its-kind analysis for SIMMC models provides useful insights and opportunities for further research in multimodal conversational agents

## introduction the situated interactive multimodal conversations (simmc) challenge 1 at dstc9 @xcite aims to lay the foundations for virtual assistant agents that can engage with the real-world, handle multimodal inputs, and perform multimodal actions. it focuses on task-oriented dialogs that encompass a situated multimodal user context in figure 1 : illustration of a simmc dialog: a user and an assistant interact in a co-observed, evolving multimodal environment for a shopping scenario. for the sake of brevity, the annotations shown are incomplete. for details of the annotation schema, see @xcite . figure adapted from @xcite . the form of a co-observed image or virtual reality (vr) environment, which is dynamically updated on each turn based on the user input and the assistant action. figure 1 illustrates an exemplary simmc dialog, where a user interacts with an assistant with the goal of browsing for furniture. here, the assistant updates the co-observed environment leading to a new multimodal context based on the dialog, e.g., visually presenting recommended chairs in a vr environment, or responding to the request "i like the brown one. show me the back of it." by executing the actions of focusing on, and rotating the indicated item. these actions in turn update the co-observed multimodal context, which grounds visual dialog @xcite image q&a n/a visual n/a n/a clevr-dialog @xcite simulated q&a n/a visual n/a n/a guesswhat @xcite image q&a n/a visual n/a n/a audio visual scene-aware dialog @xcite video q&a n/a visual n/a n/a talkthewalk @xcite image navigation visual visual + meta location u ↔ a visual-dialog navigation @xcite simulated navigation visual visual + meta location u ↔ a relative captioning @xcite image image retrieval visual visual + meta new image u ↔ a mmd @xcite image image retrieval visual visual + meta new image u ↔ a simmc @xcite image/vr task-oriented visual visual + meta situated u ↔ a + semantic the next turn of the dialog. the example highlights challenges such as multimodal action prediction (italics above) and multimodal coreference resolution (underlined elements). ## simmc challenge details we briefly review the datasets, task definitions, and evaluation used in the simmc challenge. see @xcite for additional details. datasets. two simmc datasets in the domain of interactive shopping have been provided: (1) furniture and (2) fashion. these datasets collectively contain about 13k human-to-human dialogs (totaling about 169k utterances). @xcite argue that shopping domains provide a dynamic environment, where rich multimodal interactions happen around visually grounded items. annotations. the simmc datasets are accompanied with the semantic-level annotation of utterances (dialog acts), multimodal state tracking, multimodal co-reference, actions and also ground truth semantic information about each scene. the latter allows training of virtual assistant models without the necessity of focusing on computer vision. ## related datasets and challenges compared with previous multimodal dialog datasets simmc offers four key advantages : (a) simmc assumes a co-observed multimodal context between a user and an assistant and records the ground-truth item appearance logs of each item that appears. (b) compared with the conventional taskoriented conversational datasets, the agent actions in the simmc dataset span across a diverse mul ## survey of the developed systems ## ablation study to further test the extent to which the available multimodal context is improving model results on the subtask metrics, we conduct an ablation experiment where we prepare a version of the datasets with the multimodal context removed. we then train and test the bart-large model @xcite on the original and ablated versions of the datasets. ## findings & conclusions pretrained language models show promise in multimodal settings. the strong performance of pretrained language models such as gpt-2 and bart when adapted to these task indicate their flexibility to ingest relatively simple multimodal context and thus be used in a multimodal setting with a high degree of success.