Multimodal Dialogue Response Generation (2022)

Abstract: Responsing with image has been recognized as an important capability for an intelligent conversational agent. Yet existing works only focus on exploring the multimodal dialogue models which depend on retrieval-based methods, but neglecting generation methods. To fill in the gaps, we first present a new task: multimodal dialogue response generation (MDRG) - given the dialogue history, one model needs to generate a text sequence or an image as response. Learning such a MDRG model often requires multimodal dialogues containing both texts and images which are difficult to obtain. Motivated by the challenge in practice, we consider MDRG under a natural assumption that only limited training examples are available. In such a low-resource setting, we devise a novel conversational agent, Divter, in order to isolate parameters that depend on multimodal dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of text-only dialogues and text-image pairs respectively, then the whole parameters can be well fitted using the limited training examples. Extensive experiments demonstrate our method achieves state-of-the-art results in both automatic and human evaluation, and can generate informative text and high-resolution image responses.

## introduction with the development of instant messaging technology in the recent decades, the intermediary of online conversation has also changed from pure text to a variety of visual modalities (e.g., image, gif animation, short video). similar to communicating by the messenger tools (e.g., facebook, whatsapp, wechat) in reality, an excellent intelligent conversational agent should not only be able to converse freely with plain text, but also have the ability to perceive and share the real visual physical world. although recently some large-scale pre-trained text-only dialogue generation models, such as di-alogpt @xcite , blender @xcite , meena @xcite , have shown excellent performance, they still cannot rely exclusively on plain text to completely simulate the rich experience of visual perception. recently, various vision-language tasks have been introduced and attracted widespread attention, such as visual question answering @xcite @xcite @xcite , image captioning @xcite @xcite , image-grounded dialogue @xcite @xcite @xcite . specifically, in human conversations, the images can easily show rich visual perception, which is hard to be expressed by plain text. as the example shown in figure 1 , images are required in at least three circumstances: (i) the other speaker has little knowledge (e.g., colorful burano, in the 1st image) of the objects only you had seen; (ii) to share more details (e.g., red wine and pasta, in the 2nd image) of the objects even you have common knowledge of them; (iii) to express your emotions (e.g., happy, in the 3rd image) about a specific event. an existing related task is photo sharing @xcite , which aims to select and share the image based on the textual context, is a challenging task that requires models to understand the background story which complemented by human imaginations, rather than to locate related visual objects or explicitly mention main visible content in the image as the previous works do. @xcite propose a retrieval-based method to resolve the above challenge. however, the performance of the retrieval-based method is limited in specific domains by the size of the pre-constructed conversational history repository, especially for long-tail contexts that are not covered in the history, where the set of image responses of a retrieval system is also fixed. on the other hand, a better way is to generate a new one accordingly. in this paper, we formulate a new problem: multimodal dialogue response generation (mdrg), that is, given the dialogue context, the model should not only generate a pure text response but also have the capacity to generate a multimodal response (e.g., containing both image and text). we argue that there are still some hindrances to application, since (1) the sophisticated neural end-to-end architecture will overfit to very few well-annotated training data (e.g., a few existing 10k multimodal dialogues). evidence is that when discussing the topics outside the training data domain, its performance drops dramatically; and (2) as human effort is expensive, it is not easy to collect enough training data for a new domain. based on the above facts, we take a step further to extend the assumption of mdrg to a low-resource setting where only a few multimodal dialogues are available. to tackle the above challenges, our key idea is to make parameters that rely on multimodal dialogues small and independent by disentangling textual response generation and image response generation, and thus we can learn the major part of the generation model from text-only dialogues @xmath0, @xmath1. specifically, we present divter, a novel conversational agent powered by large-scale visual world experiences. as shown in figure 2 , our divter is made up of two transformer-based @xcite components: a multimodal dialogue response generator, and a text-to-image translator. divter takes the dialogue context as input, then generates a textual sequence which may contains a text response or a textual image description or both of them. the text-to-image translator takes above image description as condition, then generates a realistic and consistent high resolution image. both components are independent with the opposite knowledge, and thus can be pre-trained using a large number of text-only dialogues and @xmath2, @xmath3. the end-to-end divter depends on the multimodal dialogues constructed as the tuple: (dialogue context, text response / <image description, @xmath4, but the joint learning and estimation of the two components just require a few training examples depending on specific domains. contributions of this work are three-fold: ## problem formalization suppose that we have dataset d @xmath5, r i )} n @xmath6, where ∀@xmath7, . . . , n}, u @xmath8,1 , . . . , u i,n i } is the dialogue context with u i,j the j-th utterance, and r i is the response regarding to u i . u i,j and r i could contain two modalities: text, and image. the goal is to learn a generation model p (r|u ; θ) (θ denotes the parameters of the model) with d s . thus, given a new dialogue context u , one can generate a multimodal response r following p (r|u ; θ). ## approach this section first formulates the unified tokenization method for multimodal dialogues. we then introduce the two important components in our proposed multimodal dialogue response generation model (divter) under low-resource scenario, including (i) textual dialogue response generator; (ii) text-to-image translator. figure 2 shows the overall of our divter. ## conclusion in this paper, we explore multimodal dialogue response generation under a low-resource setting. to overcome the challenges from the new task and insufficient training data, we propose divter, a neural conversational agent which incorporates text-toimage generation into text-only dialogue response generation, in which most parameters do not rely on the training data any more and can be estimated from large scale textual open domain dialogues @xmath9, @xmath10. extensive experiments demonstrate divter achieves state-of-the-art results in automatic and human evaluation. in the future, we will explore more efficient methods to inject more modalities into response generation.