Multimodal Context Carryover (2022)

Abstract: Multi-modality support has become an integral part of creating a seamless user experience with modern voice assistants with smart displays. Users refer to images, video thumbnails, or the accompanying text descriptions on the screen through voice communication with AI powered devices. This raises the need to either augment existing commercial voice only dialogue systems with state-of-the-art multimodal components, or to introduce entirely new architectures; where the latter can lead to costly system revamps. To support the emerging visual navigation and visual product selection use cases, we propose to augment commercially deployed voice-only dialogue systems with additional multi-modal components. In this work, we present a novel yet pragmatic approach to expand an existing dialogue-based context carryover system (Chen et al., 2019a) in a voice assistant with state-of-the-art multimodal components to facilitate quick delivery of visual modality support with minimum changes. We demonstrate a 35% accuracy improvement over the existing system on an in-house multi-modal visual navigation data set.

## related work thanks to the advent of transformer-based models over the past few years, multimodal representations have seen significant advances. the types of multimodal models can be roughly categorized into three, a) single encoder @xcite @xcite @xcite @xcite , b) dual encoder @xcite @xcite @xcite , and c) encoder-decoder models @xcite @xcite @xcite . attempts at unifying these foundational models have also been made in @xcite . single encoder models appear early in the multimodal literature and pave the way for the other 2 types of models. for this family of models, usually the image and text representations exist in separate spaces and there is an ensuing fusion layer. dual encoder models leverage image-text contrastive loss @xcite @xcite during training, exhibit higher image-to-text alignment and bring the image, text representations to a common more aligned representation space. they perform well on image-text retrieval tasks but underperform in vision-language understanding tasks requiring higher reasoning, e.g., visual question answering (vqa), and natural language inference (nli). our current task of multimodal context carryover uses the latest advances in the multimodal representation learning space and injects state-ofthe-art components with minimal changes into a framework for dialog tracking and slot selection, and results in a system that can handle multimodal user-system dialog interaction. our current multimodal use cases are set up to be similar to a textto-image retrieval task that occur within the context of a user-system dialog interaction. thus, we incorporate the latest developments in dual encoder design in our work, since the approach is well suited for the multimodal text-to-image re-trieval step. the latest state-of-the-art models that incorporate multimodal representations to dialogue state tracking systems, e.g., vdst @xcite , flamingo @xcite , and vdtn @xcite , would require costly system revamps. recently, @xcite released a novel multimodal conversation dataset with labeled dialogue state (e.g., entity and dialogue act), which motivated further studies @xcite . these datasets contain dialog act and products but do not have the scene and video information required for our purposes. for our current study we resort to collecting our own dataset through amazon mechanical turk which is catered for our commercial needs. ## conclusion we augment the existing context carryover framework with visual and vision aligned text components. we collect a multimodal dataset which mimics real world customer interactions to train and evaluate our models. we show a 35% accuracy improvement when the existing cc framework is augmented with vision and vision aligned text components.