Responsing with image has been recognized as an important capability for an intelligent conversational agent . Yet existing works only focus on exploring the multimodal dialogue models which depend on retrieval-based methods , but neglecting generation methods . To fill in the gaps , we first present a new task : multimodal dialogue response generation ( MDRG)given the dialogue context , one model needs to generate a text or an image as response . Learning such a MDRG model often requires multimodal dialogues containing both texts and images which are difficult to obtain . Motivated by the challenge in practice , we consider MDRG under a natural assumption that only limited training examples are available . Under such a low-resource setting , we devise a novel conversational agent , Divter , in order to isolate parameters that depend on multimodal dialogues from the entire generation model . By this means , the major part of the model can be learned from a large number of text-only dialogues and textimage pairs respectively , then the whole parameters can be well fitted using just a few training examples . Extensive experiments demonstrate our method achieves state-of-the-art results in both automatic and human evaluation , and can generate informative text and high-resolution image responses . With the development of instant messaging technology in the recent decades , the intermediary of online conversation has also changed from pure text to a variety of visual modalities ( e.g. , image , gif animation , short video ) . Similar to communicating by the messenger tools ( e.g. , Facebook , WhatsApp , WeChat ) in reality , an excellent intelligent conversational agent should not only be able to converse freely with plain text , but also have the ability to perceive and share the real visual physical world . Although recently some large-scale pre-trained text-only dialogue generation models , such as Di-aloGPT ( Zhang et al . , 2020 ) , Blender ( Roller et al . , 2021 ) , Meena ( Adiwardana et al . , 2020 ) , have shown excellent performance , they still can not rely exclusively on plain text to completely simulate the rich experience of visual perception . Recently , various vision-language tasks have been introduced and attracted widespread attention , such as visual question answering ( Ren et al . , 2015 ; Lu et al . , 2016 ; Anderson et al . , 2018 ; Li et al . , 2019a ; Huang et al . , 2020 ) , image captioning ( Xu et al . , 2015 ; Anderson et al . , 2016 ; Ghanimifard and Dobnik , 2019 ; Cornia et al . , 2020 ) , image-grounded dialogue ( Das et al . , 2017 ; Yang et al . , 2021 ; Agarwal et al . , 2020 ; Qi et al . , 2020 ; Chen et al . , 2021 ; Liang et al . , 2021 ) . Specifically , in human conversations , the images can easily show rich visual perception , which is hard to be expressed by plain text . As the example shown in Figure 1 , images are required in at least three circumstances : ( i ) the other speaker has little knowledge ( e.g. , colorful Burano , in the 1st image ) of the objects only you had seen ; ( ii ) to share more details ( e.g. , red wine and pasta , in the 2nd image ) of the objects even you have common knowledge of them ; ( iii ) to express your emotions ( e.g. , happy , in the 3rd image ) about a specific event . An existing related task is photo sharing ( Zang et al . , 2021 ) , which aims to select and share the image based on the textual context , is a challenging task that requires models to understand the background story which complemented by human imaginations , rather than to locate related visual objects or explicitly mention main visible content in the image as the previous works do . Zang et al . ( 2021 ) propose a retrieval-based method to resolve the above challenge . However , the performance of the retrieval-based method is limited in specific domains by the size of the pre-constructed conversational history repository , especially for long-tail contexts that are not covered in the history , where the set of image responses of a retrieval system is also fixed . On the other hand , a better way is to generate a new one accordingly . In this paper , we formulate a new problem : Multimodal Dialogue Response Generation ( MDRG ) , that is , given the dialogue context , the model should not only generate a pure text response but also have the capacity to generate a multimodal response ( e.g. , containing both image and text ) . We argue that there are still some hindrances to application , since ( 1 ) the sophisticated neural end-to-end architecture will overfit to very few well-annotated training data ( e.g. , a few existing 10k multimodal dialogues ) . Evidence is that when discussing the topics outside the training data domain , its performance drops dramatically ; and ( 2 ) as human effort is expensive , it is not easy to collect enough training data for a new domain . Based on the above facts , we take a step further to extend the assumption of MDRG to a low-resource setting where only a few multimodal dialogues are available . To tackle the above challenges , our key idea is to make parameters that rely on multimodal dialogues small and independent by disentangling textual response generation and image response generation , and thus we can learn the major part of the generation model from text-only dialogues and < image description , image > pairs that are much easier to be obtained . Specifically , we present Divter , a novel conversational agent powered by large-scale visual world experiences . As shown in Figure 2 , our Divter is made up of two Transformer-based ( Vaswani et al . , 2017a ) components : a multimodal dialogue response generator , and a text-to-image translator . Divter takes the dialogue context as input , then generates a textual sequence which may contains a text response or a textual image description or both of them . The text-to-image translator takes above image description as condition , then generates a realistic and consistent high resolution image . Both components are independent with the opposite knowledge , and thus can be pre-trained using a large number of text-only dialogues and the < image description , image > pairs respectively . The end-to-end Divter depends on the multimodal dialogues constructed as the tuple : ( dialogue context , text response / < image description , image > ) , but the joint learning and estimation of the two components just require a few training examples depending on specific domains . Contributions of this work are three-fold : • To the best of our knowledge , it is the first work on the multimodal dialogue response generation . We explore the task under a lowresource setting where only a few multimodal dialogues are assumed available . • We present Divter , a novel conversational agent which can effectively understand dialogue context and generate informative text and high-resolution image responses . • Extensive experiments on PhotoChat Corpus ( Zang et al . , 2021 ) indicate the effectiveness of Divter , it achieves a significant improvement with pure text dialogue generation model and retrieval-based image sharing method . 2 Related Work In this paper , we explore multimodal dialogue response generation under a low-resource setting . To overcome the challenges from the new task and insufficient training data , we propose Divter , a neural conversational agent which incorporates text-toimage generation into text-only dialogue response generation , in which most parameters do not rely on the training data any more and can be estimated from large scale textual open domain dialogues and < image description , image > pairs . Extensive experiments demonstrate Divter achieves state-of-the-art results in automatic and human evaluation . In the future , we will explore more efficient methods to inject more modalities into response generation .

Challenge: While responses with images are important for conversational agents, existing methods only depend on retrieval-based methods but not generation, and obtaining multimodal dialogues is difficult.
Approach: They propose a multimodal task and an agent which can learn a response generator and text-to-image translator, and use a few samples to join them.
Outcome: The proposed model achieves state-of-the-art results in automatic and human evaluation and can generate informative text and high-resolution images on PhotoChat Corpus.