Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images (2021)

Abstract: In multi-modal dialogue systems, it is important to allow the use of images as part of a multi-turn conversation. Training such dialogue systems generally requires a large-scale dataset consisting of multi-turn dialogues that involve images, but such datasets rarely exist. In response, this paper proposes a 45k multi-modal dialogue dataset created with minimal human intervention. Our method to create such a dataset consists of (1) preparing and pre-processing text dialogue datasets, (2) creating image-mixed dialogues by using a text-to-image replacement technique, and (3) employing a contextual-similarity-based filtering step to ensure the contextual coherence of the dataset. To evaluate the validity of our dataset, we devise a simple retrieval model for dialogue sentence prediction tasks. Automatic metrics and human evaluation results on such tasks show that our dataset can be effectively used as training data for multi-modal dialogue systems which require an understanding of images and text in a context-aware manner. Our dataset and generation code is available at https://github.com/shh1574/multi-modal-dialogue-dataset .

## introduction humans often use images in instant messaging services to express their meaning and intent in the dialogue context. for a dialogue system such as a chatbot to respond to human users adequately in this kind of multi-modal situations, it is necessary to understand both images and texts in their context and incorporate them in the dialogue generation process. training such a multi-modal dialogue system generally requires a large amount of training data involving images and texts in various contexts. however, numerous existing approaches relying * equal contribution. on image captioning @xcite or visual question answering @xcite techniques had to be trained with the datasets mostly irrelevant to the dialogue context. in other words, images were interpreted independently of the dialogue context, due to the lack of sufficient multi-modal dialogue datasets. those datasets containing image-grounded conversations @xcite do not even cover the situations related to dialogue context before the image, because all conversations in the dataset always start from the given image. although the relationship between images and texts can be learned using image-grounded conversations @xcite @xcite @xcite , it cannot still learn the dependency between the dialogue context before and after the image. in this paper, we propose a 45k multi-modal dialogue dataset in the form of fig. 1 . each multimodal dialogue instance consists of a textual response and a dialogue context with multiple text do you like animals? yeah, i have a cat. what's he doing now ? how are you ? fine. what are you doing ? i'm cooking now source dialogues source dialogue pre-processing processed source dialogues how are you? fine. what are you doing? i'm cooking now text-to-image replacement image source contextual-similarity-based filtering multi-modal @xmath0, i have a cat. what's he doing now ? the cat is looking at me now. = text-to-image similarity 0.57 = filter by criteria the cat is looking at me now. 0.58 0.72 0.24 0.62 raw multi-modal dialogue a b c 0.60 how are you? how are you? fine. what a.. 0.72 do you like.. yeah, i ha.. what's he.. do you like.. what's he.. the cat is... 0.24 0.58 0.62 how are you? fine. what a.. how are you? fine. what a.. 0.72 do you like.. yeah, i ha.. what's he.. do you like.. what's he.. the cat is... 0.24 0.58 0.62 fine. what a.. utterances and an image. to create this dataset, we start with existing text-only dialogue datasets as source dialogues, and then replace part of sentences in source dialogues with their semantically relevant images. the detailed steps include (1) source dialogue pre-processing, such as deleting a stop word, to improve the quality of similarity calculations, (2) creating dialogues containing an image by replacing a sentence with a similaritybased text-to-image replacement technique, and (3) pruning low-quality dialogues by employing a contextual-similarity-based filtering method. the overall process ensures that the created dataset consists of natural dialogue examples containing diverse images. in order to validate our dataset creation process and examine the quality of our multi-modal dialogue dataset, we devise the task of predicting current and next dialogue sentences while considering the dialogue context and images. we also develop simple retrieval models to learn the relationship between images and texts for the tasks. human evaluation results for predicting dialogue tasks show that the sentences are predicted as intended, i.e., in a context-aware manner, using the images. the results also show that our dataset can serve as practical training resources for multi-modal dialogue tasks that involve both image and dialogue context. ## multi-modal dialogue generation our multi-modal dialogue dataset is constructed based on three source dialogue datasets and two image captioning datasets: dailydialog @xcite , empatheticdialogues @xcite , and persona-chat @xcite for the for-mer and the ms-coco @xcite and flicker 30k @xcite for the latter. the statistics of each dataset are summarized in appendix a. after obtaining the source datasets, we replace sentences in the source dialogues with proper images by searching the image dataset to create image-mixed dialogues that maintain semantic coherence. to this end, we apply the three-stage method as shown in fig. 2: (1) source dialogue pre-processing, (2) text-to-image replacement, and (3) contextual-similarity-based filtering. ## conclusions we present the multi-modal dialogue dataset consisting of 45k multi-turn dialogues containing semantically coherent images as well as the dataset creation method. human evaluation results of our multi-modal dialogues reveal that context coherence is well maintained even if the sentence is replaced by an image, showing the validity of our dataset and data creation approach. we then evaluate our dataset using two multi-modal dialogue prediction tasks, demonstrating its effectiveness when training a dialogue system to learn the relationship between images and dialogue contexts. our proposed data creation method can be applied when efficiently preparing large-scale multi-modal dialogue datasets that cover diverse multi-modal situations. in this section, we analyze the human evaluation results for contextual-similarity-based filtering and determine thresholds for each dataset combination. the correlations between the similarity and evaluation results for each question are shown in fig. 3 . we assume that dialogue instances above the median of the evaluation score (2 for q1, q2, and 3 for q3) are suitable for use in training. based on the assumption, we determine the threshold for each combination by interpolating the median in the correlation graph of the evaluation results and the similarity. we select the largest one of three interpolated values of each question (q1, q2, and q3). the data statistics for each combination filtered by the threshold are shown in since the thresholds for each combination are determined differently, there are differences in the number of dialogue instances by combination. such results suggest that the quality of multi-modal dialogue generation may vary depending on combining the text and image datasets. for example, the dailydialog goes well with the ms-coco but not with flicker 30k. on the contrary, the empa-theticdialogues goes well with the flicker 30k but not with ms-coco. thus, we must consider finding the right combination among text and image datasets in the multi-modal dialogues generation process. we compare different module options of our model. each encoder has two options: whether to freeze or not during training, and the fusion module has two options: summation, and the attention-based transformer encoder. for final image-context fused representation, context and image representations are added in the summation fusion method, while two representations are concatenated, and then fed into the attention-based two-layer transformer encoder in the attention-based method. by this comparison, we decide to freeze only the image encoder and use the summation fusion method for both current and next dialogue prediction tasks. we additionally show the results of an information retrieval baseline, which retrieves target dialogue using the tf-idf method between candidate dialogues and the caption of an image followed by dialogue context. as shown in tables 8 and 9 , our retrieval model significantly outperforms the information retrieval baseline, indicating that comprehensive understanding of context and images is helpful in multi-modal dialogues. our implementation uses an nvida titan rtx gpu for training, and training each epoch takes about 15 minutes. our retrieval model using the summation fusion method has 204m parameters, while that using the attention-based fusion method has 254m parameters. beautiful place. i have a dog walking business. do you have pets ?