Maria: A Visual Experience Powered Conversational Agent (2021)

Abstract: Arguably, the visual perception of conversational agents to the physical world is a key way for them to exhibit the human-like intelligence. Image-grounded conversation is thus proposed to address this challenge. Existing works focus on exploring the multimodal dialog models that ground the conversation on a given image. In this paper, we take a step further to study image-grounded conversation under a fully open-ended setting where no paired dialog and image are assumed available. Specifically, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible components, i.e. , text-to-image retriever, visual concept detector and visual-knowledge-grounded response generator. The retriever aims to retrieve a correlated image to the dialog from an image index, while the visual concept detector extracts rich visual knowledge from the image. Then, the response generator is grounded on the extracted visual knowledge and dialog context to generate the target response. Extensive experiments demonstrate Maria outperforms previous state-of-the-art methods on automatic metrics and human evaluation, and can generate informative responses that have some visual commonsense of the physical world.

## introduction building intelligent conversational agents that can not only converse freely with human but also have the ability to perceive the physical world, has been one of the longest standing goals of natural language processing (nlp) and artificial intelligence (ai). although the recent large-scale conversation models trained on text-only corpora, such as meena figure 1 : an example of human conversations. when human-b talks about vacation on the beach of hawaii, human-a recalls his/her past experience of playing volleyball or having bbq on the beach. @xcite , blender @xcite and dialogpt @xcite , have shown the compelling performance, they are still lack of the perception ability to our physical world. a recent study @xcite points out the successful linguistic communication relies on a shared experience of the world that makes language really meaningful. the visual perception is a rich signal for modeling a vastness of experiences in the world that cannot be documented by text alone @xcite . on the other hand, human-human conversations involve their understandings of context, the background knowledge they had, and perhaps most importantly the experiences of the world they shared, e.g., what they have seen before. figure 1 shows a conversation between humans. human-a recalls his/her past experience of playing volleyball or having bbq on the beach when human-b talks about vacation on the beach of hawaii. however, the association relationship between beach and volleyball (or bbq) is hard to capture in traditional knowledge bases, such as knowledge graph. motivated by this, we select a common word "pizza" and collect the top 17 words that mostly co-occur with "pizza" on google knowledge graph foot_0 and ms-coco images foot_1 @xcite . as shown in figure 2 , the words cooccurring with "pizza" on knowledge graph tend to be the abstract concepts, while the co-occurrence relationship of object tags on images reflects some commonsense of our physical world, e.g., "pizza" is usually on the "dining table", people usually use "knife" when eating "pizza". interestingly, we found the "pizza" also co-occurs with "cell phone" and even "plotted plant". this indicates when people eat pizza, they sometimes would put their cell phones aside on the table, or there might exist some plotted plants in the restaurant. thus, empowering conversational agents to have the visual perception ability about the physical world is a key way for them to exhibit the human-like intelligence. the existing works @xcite @xcite focus on exploring the multimodal dialog models that ground the conversation on a given image. recently, @xcite propose to learn the dialog generation model with both image-grounded dialogs and textual dialogs by resorting to text-to-image synthesis techniques @xcite to restore a latent image for the text-only dialog. even so, these works are still constrained by the assumption that the dialog is conducted center around a given (or synthesized) image. in this paper, we take a step further to extend the assumption of image-grounded conversation to a fully open-ended setting where no imagedialog pairs are assumed available. specifically, we present maria, a neural conversational agent powered by visual world experiences which are retrieved from a pre-built image index, e.g., the open images dataset @xcite . maria consists of three components: text-toimage retriever, visual concept detector, and visualknowledge-grounded response generator. the retriever is responsible for retrieving a piece of visual world experiences, e.g., a correlated image to the dialog from an image index. the visual concept detector utilizes the object detector from updown @xcite to extract the regions features (i.e., bboxes) and the corresponding visual concepts (i.e., tags) from the retrieval images. hence, we can construct (bboxes, tags, context, response) 4-tuple as the training data. finally, these constructed 4-tuples are used to train the visualknowledge-grounded response generator, which is built on the top of a multi-layer transformer architecture @xcite . to effectively inject the visual knowledge into the response generator, we carry out the masked concept prediction and visual knowledge bias besides the response generation objective. the former aims to align the semantic representations between textual words and image regions, while the latter tries to provide more visual knowledge to facilitate the dialog generation. the experimental results on reddit conversation corpus @xcite demonstrate that maria significantly outperforms previous state-of-the-art methods, and can generate informative responses with visual commonsense of our physical world. overall, the contributions of this paper are summarized as follows: ## problem formalization suppose we have a dialog set @xmath0, r i )} n @xmath1, where ∀@xmath2, . . . , n}, c i refers to a dialog context and r i is a response to c i . we assume there is a set of images @xmath3, where ∀@xmath4, . . . , m}, v j denotes an image. ∀@xmath5, we assume that there is an image v that triggered by the given dialog context c and response r. our goal is to estimate a generation model p (r|v, c) from d and v. thus, given a new dialog context c associated with an image v , the model can generate a response r according to p (r|v, c). ## methodology to learn such a generation model p (r|v, c), we need to tackle several challenges: (1) how to bridge the gap between unpaired dialog corpus and image data; (2) after obtaining the correlated images, how to extract the detailed visual features and concepts; ## conclusions in this paper, we present maria, a neural conversational agent powered by the visual world experiences. it is able to retrieve the visual world experiences with users and generate human-like responses with some visual commonsense. extensive experiments demonstrate maria achieves substantial improvements over the state-of-the-art methods in automatic and human evaluation. the future works could include: (1) design a more precise and comprehensive image retriever to include multiple retrieval images; (2) combining the retrieve module and dialog generation into an end-to-end model, instead of learning them individually; (3) explore more efficient neural architectures to inject the visual knowledge into response generation.