Think Beyond Words: Exploring Context-Relevant Visual Commonsense for Diverse Dialogue Generation (2022)

Abstract: Commonsense knowledge has been widely considered for building intelligent open-domain dialogue agents, aiming to generate meaningful and diverse responses. Previous works in this field usually lack the ability to effectively obtain and utilize auxiliary commonsense from the external visual world. In this paper, we argue that exploiting logical information in images related to context can be effective to enrich and steer the generation process. In view of this, we propose VICTOR, a context-relevant VIsual Commonsense enhanced dialogue generaTOR for generating coherent and informative responses. To obtain the associated visual commonsense, we devise a novel approach that expands topic words on the knowledge graph and maps them into daily scenarios. During the generation, the model adopts multimodal fusion mechanism to integrate visual and textual information, and adaptively combine their decoding distributions for better response generation. The experimental results on two public datasets show that our proposed method outperforms the latest competitive methods in terms of coherence and diversity.

## introduction building intelligent dialogue systems is a longstanding goal of artificial intelligence and has attracted increasing research attention in recent years. an ideal conversation agent is supposed to generate diverse and informative responses without sacrificing their relevance to the dialogue context. to avoid general and dull dialogue generation @xcite , some approaches modify model architecture to manipulate latent variables and target distributions @xcite , yet these works limit themselves to original conversations without considering useful auxiliary information. another series of solutions augment the training corpus with extra information like emotions or per-otters dailydialog user a: how was your trip to brazil? user b: i had no idea how seriously they take soccer! visual memory visual memory figure 1 : examples from two pure language dialogue datasets, where the underlined green part is the output that needs to be generated foot_0 . the hidden visual memory, which contains associative commonsense and needs to be explored, can be essential for humans to make proper responses during the conversation. sonality @xcite . following this line, works like @xcite ; @xcite introduce more general non-conversation text like forum comments and stories to help generate richer responses. however, these works only consider information stored in pure text, ignoring the grounding information from the external visual world, which is essential for generating really meaningful language @xcite . as shown in figure 1 , it is natural that when making a conversation, we do not only focus on current context. we also expand or transition the topics by using associative memory gained from the physi-cal world, so that the chat can be more engaging and last longer. in this work, we introduce visual commonsense as the logical semantic information stored in visual scenes from daily life. considering that images representing everyday scenarios are typically logical and grounded in commonsense, it is reasonable to introduce them into open-domain conversation as additional information. @xcite ; @xcite are pioneering works that introduce visual information into the general open-domain response generation. however, these works only connect visual information by simply matching the context representation with images, without explicitly considering the topic transition of conversation. this may lead to monotonous and narrow semantics of the responses. besides, the semantic gap between modalities makes it difficult for these methods to effectively integrate visual features. furthermore, they ignore the balance between the contributions of two modalities in the decoding stage. to alleviate the above issues, we present vic-tor, a context-relevant visual commonsense enhanced dialogue generator, which consists of three components: visual commonsense retriever, multimodal fusion block, and self-adaptive response generator. the visual commonsense retriever first extracts concept words from context. then, in order to acquire explicit commonsense knowledge, it explores related concepts by multi-hop searching on knowledge graphs. each of these related concepts will be considered globally and mapped into the corresponding images, which then produce captions to narrow the semantic gap. in this way, we obtain visual commonsense with rich associative semantic information. to facilitate diverse dialogue, our multimodal fusion block incorporates auxiliary visual knowledge at each decoding step. it encodes visual commonsense with a transformer block and utilizes a coattention mechanism to fuse two modalities. the response generator is based on gpt-2 model. it takes knowledge pairs gained from knowledge graphs as guidance to encourage consistent responses with relevant topics. finally, at each decoding step, the generator uses soft probability to adaptively combine the distributions based on the textual and visual information. we demonstrate the effectiveness of our approach on two public datasets in comparison with various representative baselines. our contributions are summarized as follows: ## the proposed method in this section, we first introduce our task formulation for open-domain dialogue generation with visual commonsense, and then illustrate the three main components of our proposed victor model. ## conclusion in this work, we propose a novel context-relevant visual commonsense enhanced approach for open domain dialogue generation. the model effectively extracts relevant visual commonsense and integrates the multimodal knowledge, and adaptively measures the contribution of different modalities, so as to produce better responses. extensive experiments on two pure language dialogue datasets show that the proposed victor model significantly outperforms previous approaches, indicating that victor can generate more diverse and informative responses, while maintaining coherence with the context. for future work, we will continue to investigate the advantages of introducing external visual knowledge into the dialogue system. we notice that the current use of visual modality in this field may be too rough. further study should focus on how to extract more specific and more necessary information from images or videos for enhancing response generation. besides, enabling dialogue agents to handle multimodal inputs and outputs is also a relevant hot research problem.