Learning to Embed Multi-Modal Contexts for Situated Conversational Agents (2022)

Abstract: The Situated Interactive Multi-Modal Conversations (SIMMC) 2.0 aims to create virtual shopping assistants that can accept complex multi-modal inputs, i.e. visual appearances of objects and user utterances. It consists of four subtasks, multi-modal disambiguation (MM-Disamb), multi-modal coreference resolution (MM-Coref), multi-modal dialog state tracking (MM-DST), and response retrieval and generation. While many task-oriented dialog systems usually tackle each subtask separately, we propose a jointly learned multi-modal encoder-decoder that incorporates visual inputs and performs all four subtasks at once for efficiency. This approach won the MM-Coref and response retrieval subtasks and nominated runner-up for the remaining subtasks using a single unified model at the 10th Dialog Systems Technology Challenge (DSTC10), setting a high bar for the novel task of multi-modal task-oriented dialog systems.

## introduction a task-oriented dialog system aims to assist users to accomplish certain tasks, such as executing actions or retrieving specific information, with natural language conversations. with the rising interest in multi-modal representation learning, the next generation of task-oriented virtual assistants is expected to handle conversations in such contexts, especially in the domain of vision-language (vl). for instance, a multi-modal dialog agent may help the user navigate a virtual clothing store and look for an object meeting the user's criteria. in such cases, a successful dialog agent should be able to parse and understand multi-modal contexts. to this end, simmc 2.0 @xcite proposes a situated multi-modal context in the form of co-observed, realistic scene set in virtual reality (vr) stores to incorporate the complexity of multi-modal task-oriented dialogs. the multimodal subtasks, mm-disamb and mm-coref, intend to test the assistant's capability to identify the need for disambiguating reference mentions and to ground them to the scene objects. while challenging, these are all essential to building a successful multi-modal dialog agent. in this paper, we present our end-to-end, jointlearning approach to address this challenge in simmc 2.0. we adopt bart @xcite and attach task-specific heads so that the model can make predictions on all subtasks at once. to be more specific, our model performs mm-disamb, mm-coref, and response retrieval by the encoder and mm-dst and response generation in a string format by the decoder. we also integrate multimodality into the model by extracting visual features of each object from a convolutional vision backbone and then combining them with nonvisual attributes. our model is jointly trained on all subtasks and a couple of auxiliary objectives to help the model align the different modalities. for retrieval, we use in-batch negative samples for contrastive metric learning instead of creating a pool of separate training samples. with modification for the competition setting, our model was ranked at the first place for mm-coref and response retrieval with 75.8% coreference f1, 82.5% mrr, 72.5% r@1, 95.0% r@5, 98.4% r@10, and 1.9 mean rank in the official evaluation of dstc10. moreover, our model was nominated runner-up for all other subtasks, in which we achieved 93.8% disambiguation accuracy, 90.3% slot f1, 95.9% intent f1, and 0.295 bleu-4. the results were obtained with only a single model and consistent with the results on the devtest (i.e. validation) set, demonstrating a robust, common representation on all subtasks learned by the model. ## related work recent works on (uni-modal) task-oriented dialog systems remove the need for a pipeline composed of nlu @xcite , dst @xcite , pol @xcite , and nlg @xcite modules by leveraging pretrained language models (lm) that integrate all the modules in an end-to-end, auto-regressive manner @xcite @xcite . given a dialog context, such systems sequentially generates belief state, system action, and response, making predictions based on decisions made by previous modules in the form of tokens and achieving superior results to the pipelined approaches. some of these systems aim to learn the user preference from dialogs and recommend the object based on external knowledge base (kb) @xcite . in a similar context, building cross-modal models has recently gained attention in vl domain. recent works develop vl models on top of the transformer-based @xcite pretrained lm and vision backbones, focusing on self-supervised pretraining methods to align joint embedding between different modalities. they achieve state-of-the-art performance in downstream tasks such as visual question answering (vqa), as shown in @xcite and @xcite . however, there are only a handful of works focusing on situated vl task-oriented dialog systems @xcite , where visual modality of the task is provided in a sanitized setting rather than a natural, situated scene. ## simmc 2.0 description the simmc 2.0 dataset consists of 11,244 dialogs split into train (65%), dev (5%), devtest (15%), and teststd (15%) sets. each dialog includes multiple turns where each turn has grounded multi-figure 1 : an instance of dialog and the corresponding scene in simmc 2.0. here, the assistant asks the user to disambiguate between the blue hoodie jacket (denoted 1) and the beige jacket (denoted 2), grounding its mentions to the scene via multi-modal context m 0 = {1, 2}. once the user chooses the blue one, the system retrieves the information on the disambiguated object. the multimodal context in this case would be m 1 = {1}. the assistant needs to make predictions conditioned on history h t , current user utterance u t , and the scenes up to the current turn s @xmath0. the object set consists of fashion and furniture domain, where each domain has 288 and 57 items respectively. the system is allowed to look up which item is present in a scene at all time along with its bounding box information. as a side information, the metadata of each object are provided: its non-visual attributes such as brand, size, customer rating and price are available for both training and inference, but looking up the visual attribute (e.g. color, pattern, materials, sleeve length) is prohibited for inference so as to make the agent reason with multi-modal information. figure 2 : overview of the jointly learned multi-tasking bart. for h t , we show only the last turn without user utterance due to space limit. the details on the loss functions are provided in model specifics. each scene object is represented by the concatenation of scene canonical object id token (e.g. <11>) and features from a vision encoder. it is then passed through mm-coref and attribute classification head. mm-dst and response generation subtasks are approached in terms of auto-regressive lm. ## multi-modal transformer model the setting of the dataset is similar to that of vqa where finetuning the pretrained vl models is prevalent; however, these models are usually pretrained on natural images @xcite and require a large number of training samples of 3d rendered images that are aligned properly with text. hence, we decide to work primarily with pretrained lm and convolutional vision en-coder to suit the setting of simmc 2.0. in particular, we integrate the visual modality by encoding each object with finetuned resnet-34 @xcite . we also index each object in the scene by its referential id (canonical object id), which are concatenated with corresponding visual representations for subtasks. in order to further align the different modalities, we provide additional supervision signals at train time by looking up the object metadata. we note that all of the subtasks are related to each other. for example, if the assistant decides that the user utterance needs to be disambiguated, then the appropriate system action is to respond along the line of "which one are you referring to?". once disambiguated, the user may ask for the price of "blue striped shirt", where representations learned from mm-coref prediction subtask (and/or attribute classification) can help the model predict the correct slot values for mm-dst and response generation. we expect that the latent representation of the multimodal dialog learned from other subtasks will translate readily to other subtasks. hence, we utilize hard parameter sharing @xcite on the encoder to jointly learn on all subtasks. this reduces not only the number of network parameters, but also the risk of overfitting @xcite . moreover, we decide to view mm-coref as a type of set prediction @xcite , where joint learning of set cardinality and state distribution has been shown effective @xcite . hence, we define an additional empty coreference target prediction (empty-coref), a simplified cardinality prediction task that outputs whether the current user utterance has no mm-coref targets. moreover, we perform supervised learning on object attributes to help align object-language modalities. we adopt bart @xcite as the pretrained language backbone. our preliminary experiments suggested that performing certain subtasks such as mm-disamb by a bi-directional encoder (e.g. bert) proved more effective than doing so by an uni-directional decoder (e.g. gpt-2). to harness both the nlu capabilities of the encoder and the nlg capabilities of the decoder, we choose a transformer encoder-decoder to handle all subtasks at once. we attach classification heads for mm-disamb and mm-coref subtasks to the encoder and lm head for mm-dst and response generation to the decoder. we also perform retrieval by computing the dot product between representation vectors of response candidates and multi-modal dialog context. figure 2 provides an overview of the model. ## results the results on the devtest (validation) and teststd (test) splits are shown in for comparison, the winning entry for mm-disamb and mm-dst, entry #5, uses separate models, namely roberta-large @xcite for mm-disamb and bart for generating mm-coref and mm-dst using the same prompt of the baselines without the use of visual features. even though injecting continuous visual features (as extracted by vision models) may introduce noise for generation, they certainly help with mm-coref subtasks as some entries achieving more than 65% object f1 utilize visual features (#6 and #7). entry #8 enumerate visual attributes in the form of natural language tokens without relying on actual visual features. entry #6 (lee and han, 2021) is a multi-tower architecture with text encoder (roberta-large) and image encoder (deit) @xcite . to adopt the image encoder to the simmc 2.0 domain, it is contrastively pretrained by matching object image to its natural language attributes and scene (background) image to dialog context. then, the objects and scene representations are added together to match against the dialog context for mm-coref prediction. entry #7 @xcite encodes object information (index, location, and image) extracted by clip @xcite and butd @xcite then inputs the flattened object representations to uniter @xcite along with dialog context and scene image. mm-coref predictions are made in terms of binary classification, similar to our approach. all of the response retrieval entries modify the baseline approach, where the generated response (not the dialog context) is compared against the response candidate pool by different measures. entry #1 uses cosine similarity for retrieval score instead of cross-entropy. entry #2 uses negative likelihood, but generates from bart. entry #4 follows os-car @xcite with self-supervised few-shot learning for predicting object tags, which act as an anchor between image (object) and text (dialog) modalities. the generated response with attached decoder is then compared in the same way as the baseline. ## conclusion in this paper, we propose a multi-modal taskoriented dialog system based on bart that can perform all simmc 2.0 subtasks at once. our model integrates the multi-modality by utilizing features from a vision model. in addition to the joint learning of all subtasks, we introduce auxiliary tasks. we observe that the joint-learning and other components are crucial in building a successful multi-modal assistant for simmc 2.0. our model is able to perform competitively in all of the subtasks, setting a high bar for the new generation of multi-modal task-oriented dialog systems. despite the success in simmc 2.0, our approach has a few limitations. first, it relies on metadata for non-visual attributes, which may not generalize well if a new set of domain items are introduced at inference. our method also fails to fully capture the locality of objects within the scene (e.g. on the table, in the closet, etc.). we believe that these limitations will be addressed in future works.