Multimodal Conversational AI : A Survey of Datasets and Approaches (2022)

Abstract: As humans, we experience the world with all our senses or modalities (sound, sight, touch, smell, and taste). We use these modalities, particularly sight and touch, to convey and interpret specific meanings. Multimodal expressions are central to conversations; a rich set of modalities amplify and often compensate for each other. A multimodal conversational AI system answers questions, fulfills tasks, and emulates human conversations by understanding and expressing itself via multiple modalities. This paper motivates, defines, and mathematically formulates the multimodal conversational research objective. We provide a taxonomy of research required to solve the objective: multimodal representation, fusion, alignment, translation, and co-learning. We survey state-of-the-art datasets and approaches for each research area and highlight their limiting assumptions. Finally, we identify multimodal co-learning as a promising direction for multimodal conversational AI research.

## introduction the proliferation of smartphones has dramatically increased the frequency of interactions that humans have with digital content. these interactions have expanded over the past decade to include conversations with smartphones and in-home smart speakers. conversational ai systems (e.g., alexa, siri, google assistant) answer questions, fulfill specific tasks, and emulate natural human conversation @xcite . early examples of conversational ai include those based on primitive rule-based methods such as eliza @xcite . more recently, conversational systems were driven by statistical machine translation systems: translating input queries to responses @xcite . orders of magnitude more data led to unprecedented advances in conversational technology in the mid-part of the last decade. tech-niques were developed to mine conversational training data from the web search query-click stream @xcite @xcite and web-based knowledge graphs @xcite . with this increase in data, deep neuralnetworks gained momentum in conversational systems @xcite @xcite @xcite @xcite . most recently, specialized deep learning-based conversational agents were developed primarily for three tasks: (1) goal-directed tasks in research systems @xcite @xcite @xcite @xcite @xcite and commercial products (siri, cortana, alexa, and google assistant), (2) questionanswering @xcite @xcite , and (3) open-domain conversations @xcite @xcite @xcite @xcite . however, developing a single system with a unified approach that achieves human-level performance on all three tasks has proven elusive and is still an open problem in conversational ai. one limitation of existing agents is that they often rely exclusively on language to communicate with users. this contrasts with humans, who converse with each other through a multitude of senses. these senses or modalities complement each other, resolving ambiguities and emphasizing ideas to make conversations meaningful. prosody, auditory expressions of emotion, and backchannel agreement supplement speech, lip-reading disambiguates unclear words, gesticulation makes spatial references, and high-fives signify celebration. alleviating this unimodal limitation of conversational ai systems requires developing methods to extract, combine, and understand information streams from multiple modalities and generate multimodal responses while simultaneously maintaining an intelligent conversation. similar to the taxonomy of multimodal machine learning research @xcite , the research required to extend conversational ai systems to multiple modalities can be grouped into five areas: representation, fusion, translation, alignment, and co-learning. representation and fusion involve learning mathematical constructs to mimic sensory modalities. translation maps relationships between modalities for cross-modal reasoning. alignment identifies regions of relevance across modalities to identify correspondences between them. co-learning exploits the synergies across modalities by leveraging resourcerich modalities to train resource-poor modalities. concurrently, it is necessary for the research areas outlined above to address four main challenges in multimodal conversational reasoning -disambiguation, response generation, coreference resolution, and dialogue state tracking @xcite . multimodal disambiguation and response generation are challenges associated with fusion that determine whether available multimodal inputs are sufficient for a direct response or if follow-up queries are required. multimodal coreference resolution is a challenge in both translation and alignment, where the conversational agent must resolve referential mentions in dialogue to corresponding objects in other modalities. multimodal dialogue state tracking is a holistic challenge across research areas typically associated with task-oriented systems. the goal is to parse multimodal signals to infer and update values for slots in user utterances. in this paper, we discuss the taxonomy of research challenges in multimodal conversational ai as illustrated in figure 1 . section 2 provides a history of research in multimodal conversations. in section 3, we mathematically formulate multimodal conversational ai as an optimization problem. sections 4, 5, and 6 survey existing datasets and state-of-the-art approaches for multimodal representation and fusion, translation, and alignment. section 7 highlights limitations of existing research in multimodal conversational ai and explores multimodal co-learning as a promising direction for research. image text video audio scene mask dialogue speech action [h 11 , h 12 , ...... h 1n ] t [h 21 , h 22 , ...... h 2n ] t [h 31 , h 32 , ...... h 3n ] t [h m1 , h m2 , ...... h mn ] t multimodal fusion multimodal representation learning image embedder text embedder video embedder audio embedder [z 11 , z 12 , ...... z 1n ] t [z 21 , z 22 , ...... z 2n ] t [z 31 , z 32 , ...... z 3n ] t [z m1 , z m2 , ...... z mn ] t multimodal alignment multimodal alignment multimodal alignment multimodal translation multimodal translation multimodal co-learning ..... ..... 2 background early work in multimodal conversational ai focused on the use of visual information to improve automatic speech recognition (asr). one of the earliest papers along these lines is by @xcite followed by many papers including work by @xcite , @xcite , @xcite , and @xcite . advances in client-side capabilities enabled asr systems to utilize other modalities such as tactile, voice, and text inputs. these systems supported more comprehensive interactions and facilitated a higher degree of personalization. examples include esprit's mask @xcite , microsoft's mipad @xcite , and at&t's match @xcite . vision-driven tasks motivated research in adding visual understanding technology into conversational ai systems. early work in reasoning over text+video include work by @xcite where they leveraged these combined modalities to address the problem of assigning names of people in the cast to tracks in tv videos. @xcite leveraged natural language descrptions of rgb-d videos for 3d semantic parsing. @xcite developed a multimodal deep boltzmann machine for image-text retrieval and asr using videos. @xcite introduced a dataset and baselines for multimodal question-answering, a challenge combining computer vision and natural language pro-cessing. more recent work by @xcite and @xcite leveraged conversational explanations to make vision and language models more grounded, resulting in improved visual question answering. while modalities most commonly considered in the conversational ai literature are text, vision, tactile, and speech, other sources of information are gaining popularity within the research community. these include eye-gaze, 3d scans, emotion, action and dialogue history, and virtual reality. @xcite al. (2013) processing conventional and new modalities brings forth numerous challenges for multimodal conversations. to answer these challenges, we will first mathematically formulate the multimodal conversational ai problem, then detail fundamental research sub-tasks required to solve it. ## mathematical formulation we formulate multimodal conversational ai as an optimization problem. the objective is to find the optimal response s to a message m given underlying multimodal context c. based on the sufficiency of the context, the optimal response could be a statement of fact or a follow-up question to resolve ambiguities. statistically, s is estimated as: the probability of an arbitrary response r can be expressed as a product of the probabilities of responses {r i } t @xmath0. it is also possible for conversational ai to respond through multiple modalities. we represent the multimodality of output responses by a matrix r := {r 1 i , r 2 i , . . . r l i } over l permissible output modalities. learning from multimodal data requires manipulating information from all modalities using a function f (•) consisting of five sub-tasks: representation, fusion, translation, alignment, and co-learning. we include these modifications and present the final multimodal conversational objective below. in the following sections, we describe each subtask contained in f (•). ## multimodal representation + fusion multimodal representation learning and fusion are primary challenges in multimodal conversations. multimodal representation is the encoding of multimodal data in a format amenable to computational processing. multimodal fusion concerns joining features from multiple modalities to make predictions. ## multimodal translation multimodal translation maps embeddings from one modality to signals from another for cross-modality reasoning (figure 1 ). cross-modal reasoning enables multimodal conversational ai to hold meaningful conversations and resolve references across multiple senses, specifically language and vision. to this end, we survey existing work addressing the translation of images and videos to text. we discuss multimodal question-answering and multimodal dialogue, translation tasks that extend to multimodal conversations. @xcite and @xcite present visual question-answering (vqa) and visual7w for multimodal question answering (mqa). the mqa challenge requires responding to textual queries about an image. both datasets collect questions and answers using crowd workers, encouraging trained models to learn natural responses. @xcite presents the visual slot dataset, where trained models learn answers to questions grounded in uis. ## multimodal alignment while image-based dialogue revolves around objects (e.g., cats and dogs), video-based dialogue revolves around objects and associated actions (e.g., jumping cats and barking dogs) where spatial and temporal features serve as building blocks for conversations. extracting these spatiotemporal features requires multimodal alignment -aligning sub-components of different modalities to find correspondences. we identify action recognition and action from modalities as alignment challenges relevant to multimodal conversations. ## discussion the current datasets used for research in multimodal conversational ai are summarized in similarly, the work to-date on action recognition is incomplete and insufficient for conversational systems. conversational ai must represent and understand spatiotemporal interactions. however, current research in action recognition attempts to learn relationships between videos and their natural language descriptions. these descriptions are not speech acts themselves. therefore, they do not adequately represent dialogue but rather only serve as anchor points in the interaction. in contrast, image-chat (shuster et al., 2020) presents a learning challenge directly aligned with the multimodal dialogue objective in equation 4. image-chat treats dialogue as an open-ended discussion grounded in the visual modality. succeeding in the task requires jointly optimizing visual and conversational performance. the use of crowd workers that adopt personalities during data collection encourages natural dialogue and captures conversational intricacies and implicatures. mqa answers explicit questions about an image ( →is this at a farm?), and action recognition describes videos ( →mountain biking). on the other hand, image-chat requires both implicit knowledge ( → halloween, → exercise) and multi-turn reasoning ( → @xmath1, → @xmath2. despite its advantages over other datasets, image-chat makes three assumptions about multimodal conversations limiting its extension to the multimodal conversational objective: void of long-term dialogue dependencies. the discussion above highlights the limitations of existing datasets for the multimodal conversational ai task. datasets need to be improved to better capture and represent more natural, multiturn dialogues over multiple modalities; dialogues that more closely resemble how humans converse with each other and their environment. in addition, algorithmic improvements are required to advance the field of multimodal conversational ai -particularly with respect to the objective function. current approaches such as mqa and action recognition models optimize a limited objective compared to equation 4. we postulate that the degradation of these methods when applied to multimodal conversations is largely caused by this and, therefore, motivates investigation. another open research problem is to improve performance on image-chat. the current state-ofthe-art transresnet ret is limited. the model often hallucinates, referring to content missing in the image and previous dialogue turns. the model also struggles when answering questions and holding extended conversations. we suspect these prob-lems are a reflection of the limiting assumptions image-chat makes and the absence of multimodal co-learning to extract relationships between modalities. for further details, we refer readers to example conversations in appendix a. different modalities often contain complementary information when grounded in the same concept. multimodal co-learning exploits this crossmodality synergy to model resource-poor modalities using resource-rich modalities. an example of co-learning in context of figure 1 is the use of visual information and audio to generate contextualized text representations. @xcite introduced an early approach to multimodal co-training, using information from hyperlinked pages for web-page classification. socher and fei-fei (2010 ## conclusions we define multimodal conversational ai and outline the objective function required for its realization. solving this objective requires multimodal representation and fusion, translation, and alignment. we survey existing datasets and state-ofthe-art methods for each sub-task. we identify simplifying assumptions made by existing research preventing the realization of multimodal conversational ai. finally, we outline the collection of a suitable dataset and an approach that utilizes multimodal co-learning as future steps.