Knowledge Transfer with Visual Prompt in multi-modal Dialogue Understanding and Generation (2022)

Abstract: Visual Dialogue (VD) task has recently received increasing attention in AI research. Visual Dialog aims to generate multi-round, interactive responses based on the dialog history and image content. Existing textual dialogue models cannot fully understand visual information, resulting in a lack of scene features when communicating with humans continuously. Therefore, how to efficiently fuse multimodal data features remains to be a challenge. In this work, we propose a knowledge transfer method with visual prompt (VPTG) fusing multi-modal data, which is a flexible module that can utilize the text-only seq2seq model to handle visual dialogue tasks. The VPTG conducts text-image co-learning and multi-modal information fusion with visual prompts and visual knowledge distillation. Specifically, we construct visual prompts from visual representations and then induce sequence-to-sequence(seq2seq) models to fuse visual information and textual contexts by visual-text patterns. And we also realize visual knowledge transfer through distillation between two different models’ text representations, so that the seq2seq model can actively learn visual semantic representations. Extensive experiments on the multi-modal dialogue understanding and generation (MDUG) datasets show the proposed VPTG outperforms other single-modal methods, which demonstrate the effectiveness of visual prompt and visual knowledge transfer.

## introduction cross-modal understanding between vision and language has become a challenging field in natural language processing and computer vision. with the rapid development of deep neural networks, researchers have made rapid progress in a series of visual language tasks, including moment localization with natural language @xcite @xcite , image figure 1 : description of the multi-modal dialogue understanding and generation (mdug) task. from step1 to step 3, the video is about a priest, and the subtitles are snippets of wedding vows. for the response generation of step 4, supposing that only dialogue text context was taken, the previous dialog text: "ok, then" is inadequate for generating the expected output: "you may kiss the bride." captioning @xcite @xcite , visual question answering @xcite @xcite , etc. the visual dialogue task @xcite aims to perform multiple rounds of interactive dialogue based on dialogue history and image content. dialogues with multi-modal contexts (visual and textual) are becoming more and more general in daily life @xcite , such as communicating messenger tools (e.g. facebook, wechat). compared with visual question answering, visual dialogue (vd) tasks not only require answering questions according to visual information but also require a deep understanding of multiple rounds of historical dialogues @xcite @xcite . in the visual dialogue task, researchers have put forward a lot of relevant datasets, the guesswhat?! @xcite and the visdial @xcite set up visual dialog data sets for images. the mdug @xcite is based on video scenes to generate coherent textual responses. in this work, we mainly focus on video visual dialogue such as the multi-modal dialogue understanding and generation (mdug) dataset @xcite . compared to image captioning and image visual dialogue, it requires modeling longdistance image sequences, which is more challenging and practical. the mdug task proposes a multi-modal dialogue task in the video field. it needs the system to generate a response of the current frame based on multi-modal video scene and historical dialogue information, where historical video clips frame and text captions are mapped one-to-one. the video clips and visual images have much abundant and useful information about the plot development. it is easy to pick up on their movements and expressions from visual information. for example, in the last frame of figure 1 . on the one hand, from the body movements of people such as they gradually face each other and a smile on the man's face, we can observe that the man is going to kiss his bride, so models can infer the "kiss" action in generated response. on the other hand, from the wedding vows context, it's easy to infer their roles as bride and groom. therefore, this example demonstrates the importance of combining images and texts for the mdug task. although much attention has been drawn to dialogue tasks @xcite , neural models have shown impressive performance gains in textual dialogue tasks. but existing text-only dialogue methods still have limitations in handling video dialogue tasks in multi-modal scenarios, which may hinder further advancement in this direction. in textonly dialogue tasks, more and more text generation models are pre-trained in the large-scale corpora with the development of pre-trained language models @xcite . most of the dialogue pre-training models are based on transformers through pre-training in large-scale dialogue texts and using a large number of encoder and decoder layers @xcite @xcite . this can improve the consistency between the generated context and context and the fluency of the generated text. but the bigger challenge is based on the non-homogeneity of the input text-image multi-modal information and the output text information besides challenges in the text-only task in multi-modal dialogue generation tasks. how to understand and integrate the multi-modal information, and comprehensively perform text generation remains to be an unsolved and important problem. many efforts have been made to realize a reliable and accurate multi-modal dialogue understanding and generation in similar tasks such as image captioning and video question answering @xcite @xcite . however, the methods adopted in that work cannot be directly generalized to the video visual dialogue task, and the video visual dialogue task requires multi-level modeling in a large number of sequence images and dialog history at the same time @xcite . to take a significant step in this direction and fully utilize seq2seq models' capability, we propose a visual prompt text generate (vptg) method that can directly provide visual assistance training for multi-modal language models to tackle the above challenges. the vptg framework can efficiently generate dialogue response that is coherent to both visual images and text dialogue. to model text-image mapping in the same representation space, we adopt clip contrastive training to conduct co-learning of image-caption pairs through a pre-trained language model @xcite . we also use the visual prompt to fuse image visual information into text features. in the training stage, we input the "image" and "answer text" into the clip @xcite , and input the "image" feature vector as a visual prompt into the seq2seq model. in addition, to improve the visual modeling ability of language models, we conduct visual knowledge transfer by transferring visual representations to visual prompt and using it to prompt the seq2seq model modeling multi-modal data. specifically, the "answer text" feature is also provided to the encoder output "[cls]" vector of the seq2seq model for distillation. we also ask the sequence-tosequence (seq2seq) model to actively learn visual semantic representations. for efficient training, we adopt an end-to-end training architecture. in the prediction stage, we only use the image as the input of the clip and get the visual prompt, and then perform multi-level learning from visual information to textual information. in the vptg, we perform efficient representation, co-learning, and fusion of multi-modal information. extensive experimental results show that the vptg method consistently outperforms all baseline schemes in the mdug task, showing the effective ability of the method to make better use of textual and visual information to generate high-quality multi-modal dialogue responses. in summary, our contributions are as follows: ## datasets the multi-modal dialogue understanding and generation task @xcite ) is required to generate a dialogue agent for the next sentence based on the multi-modal scene and the previous dialogue process. this task needs to model the semantics of the session and the scenario of the session. the task provides the multi-modal video of dialogue content and scene. its ultimate goal is to generate agent replies that meet the context and are related to the video scene. the videos and dialogues for this task are crawled from online tv series. the dataset is split into a training set, a validation set, and a test set. each example includes a dialogue session as well as the associated video clip, which is a sequence of frames. the frames from the videos have been downsampled to 3fps. it is composed of 43,895 videos with 1,100,242 utterances. each video has an average of 25.07 utterances. we follow the official data split, where 1,000,079, 50,032, and 50,131 utterances are used for training, validation, and testing, respectively. ## the proposed method we propose the visual prompt text generate (vptg) framework for the multi-modal dialogue understanding and generation (mdug) task, whose ultimate goal is to generate a response that is coherent to the dialogue context and relevant to the video context. the figure 2 illustrates the architecture of vptg. it is challenging to directly generate the dialogue response according to multi-modal data. to tackle this challenge of data alignment and fusion between image and text, we split the mdug task into two simultaneous modules: (1) the visual predictor module is first used to generate visual prompt (section 4.1) by jointly training an image encoder and a text encoder and fusion image information into a text representation. (2) the text predictor conducts visual knowledge transfer (section 4.2) to guarantee response generation with information alignment between text and image. ## experiments in this section, we will introduce the evaluation indicators and experimental settings. then we compare vptg with the existing dialogue generation technology and ablation experiments to prove the effectiveness of our method. ## conclusions in this paper, we proposed a new visual knowledge fusing paradigm that provides the pre-trained language generation model with the visual prompt. the vptg module is flexible and can support almost all seq2seq models to be used in multimodal dialogue generation tasks. it realizes the language model's understanding of visual infor-mation by transforming visual features into embedding prompts. we have conducted vast experiments on the task of multi-modal dialogue understanding and generation. the vptg outperforms all other baselines in mdug tasks for these experiments, which reflects the effectiveness of the proposed method.