Multimodal Dialogue State Tracking (2022)

Abstract: Designed for tracking user goals in dialogues, a dialogue state tracker is an essential component in a dialogue system. However, the research of dialogue state tracking has largely been limited to unimodality, in which slots and slot values are limited by knowledge domains (e.g. restaurant domain with slots of restaurant name and price range) and are defined by specific database schema. In this paper, we propose to extend the definition of dialogue state tracking to multimodality. Specifically, we introduce a novel dialogue state tracking task to track the information of visual objects that are mentioned in video-grounded dialogues. Each new dialogue utterance may introduce a new video segment, new visual objects, or new object attributes and a state tracker is required to update these information slots accordingly. We created a new synthetic benchmark and designed a novel baseline, Video-Dialogue Transformer Network (VDTN), for this task. VDTN combines both object-level features and segment-level features and learns contextual dependencies between videos and dialogues to generate multimodal dialogue states. We optimized VDTN for a state generation task as well as a self-supervised video understanding task which recovers video segment or object representations. Finally, we trained VDTN to use the decoded states in a response prediction task. Together with comprehensive ablation and qualitative analysis, we discovered interesting insights towards building more capable multimodal dialogue systems.

## introduction the main goal of dialogue research is to develop intelligent agents that can assist humans through conversations. for example, a dialogue agent can be tasked to help users to find a restaurant based on their preferences of price ranges and food choices. a crucial part of a dialogue system is dialogue state tracking (dst), which is responsible for tracking and updating user goals in the form of dialogue states, including a set of (slot, value) pairs such as (price, "moderate") and (food, "japanese"). numerous machine learning approaches have been proposed to tackle dst, including fixed-vocabulary models @xcite and open-vocabulary models @xcite @xcite , for either single-domain @xcite or multi-domain dialogues @xcite . however, the research of dst has largely limited the scope of dialogue agents to unimodality. in this setting, the slots and slot values are defined by the knowledge domains (e.g. restaurant domain) and database schema (e.g. data tables for restaurant entities). the ultimate goal of dialogue research towards building artificial intelligent assistants necessitates dst going beyond unimodal systems. in this paper, we propose multimodal dialogue state tracking (mm-dst) that extends the dst task in a multimodal world. specifically, mm-dst extends the scope of dialogue states by defining slots and slot values for visual objects that are mentioned in visually-grounded dialogues. for research purposes, following @xcite , we limited visually-grounded dialogues as ones with a grounding video input and the dialogues contain multiple turns of (question, answer) pairs about this video. each new utterance in such dialogues may focus on a new video segment, new visual objects, or new object attributes, and the tracker is required to update the dialogue state accordingly at each turn. an example of mm-dst can be seen in figure 1 . toward mm-dst, we developed a synthetic benchmark based on the cater universe @xcite . we also introduced video-dialogue transformer network (vdtn), a neural network architecture that combines both objectlevel features and segment-level features in video and learns contextual dependencies between videos ## multimodal dialogue state tracking traditional dst. as defined by @xcite , the traditional dst includes an input of dialogue d and a set of slots s to be tracked from turn to turn. at each dialogue turn t, we denote the dialogue context as d t , containing all utterances up to the current turn. the objective of dst is for each turn t, predict a value v t i of each slot s i from a predefined set s, conditioned by the dialogue context d t . we denote the dialogue state at turn motivation to multimodality. yet, the above definition of dst are still limited to unimodality and our ultimate goal of building intelligent dialogue agents, ideally with similar level of intelligence as humans, inspires us to explore mulitmodality. in neuroscience literature, several studies have analyzed how humans can perceive the world in visual context. @xcite found that humans can recognize multiple visual objects and how their contexts, often embedded with other related objects, facilitate this capacity. our work is more related to the recent study @xcite which focuses on human capacity to create temporal stability across multiple objects. the multimodal dst task is designed to develop multimodal dialogue systems that are capable of maintaining discriminative representations of visual objects over a period of time, segmented by dialogue turns. while computer science literature has focused on related human capacities in intelligent systems, they are mostly limited to vision-only tasks e.g. @xcite or qa tasks e.g. @xcite but not in a dialogue task. most related work in the dialogue domain is @xcite and almost concurrent to our work is @xcite . however, @xcite is limited to a single object per dialogue, and @xcite extends to multiple objects but does not require to maintain an information state with component slots for each object. our work aims to complement these directions and address their limitations with a novel definition of multimodal dialogue state. ## video-dialogue transformer network a naive adaptation of conventional dst to mm-dst is to directly combine visual features extracted by a pretrained 3d-cnn model. however, as ## discussion and conclusion compared to conventional dst @xcite @xcite , we show that the scope of dst can be further extended to a multimodal world. compared to prior work in multimodal dialogues @xcite @xcite which focuses more on vision-language interactions, our work was inspired from a dialogue-based strategy with a formulation of a dialogue state tracking task. for more comparison to related work, please refer to appendix a. we noted the current work are limited to a synthetic benchmark with a limited video domain (3d objects). however, we expect that mm-dst task is still applicable and can be extended to other video domains (e.g. videos of humans). we expect that mm-dst is useful in dialogues centered around a "focus group" of objects. for further discussion of limitations, please refer to appendix e. in summary, in this work, we introduced a novel mm-dst task that tracks visual objects and their attributes mentioned in dialogues. for this task, we experimented on a synthetic benchmark with videos simulated in a 3d environment and dialogues grounded on these objects. finally we proposed vdtn, a transformer-based model with self-supervised learning objectives on object and segment-level visual representations.