D ialog CC : An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset (2024)

Abstract: As sharing images in an instant message is a crucial factor, there has been active research on learning an image-text multi-modal dialogue models.However, training a well-generalized multi-modal dialogue model remains challenging due to the low quality and limited diversity of images per dialogue in existing multi-modal dialogue datasets.In this paper, we propose an automated pipeline to construct a multi-modal dialogue dataset, ensuring both dialogue quality and image diversity without requiring minimum human effort. In our pipeline, to guarantee the coherence between images and dialogue, we prompt GPT-4 to infer potential image-sharing moments - specifically, the utterance, speaker, rationale, and image description. Furthermore, we leverage CLIP similarity to maintain consistency between aligned multiple images to the utterance.Through this pipeline, we introduce DialogCC, a high-quality and diverse multi-modal dialogue dataset that surpasses existing datasets in terms of quality and diversity in human evaluation.Our comprehensive experiments highlight that when multi-modal dialogue models are trained using our dataset, their generalization performance on unseen dialogue datasets is significantly enhanced. We make our source code and dataset publicly available (https://dialogcc.github.io/).

## introduction people share various images with each other when communicating via instant messaging tools. such behavior increases social bonding (rapport) as well as engagement. the ability to share images is also necessary for a dialogue model for better bonding conversations. in the visual dialogue domain, the majority of previous works have focused on image-grounded dialogues, where two persons talk about given images @xcite @xcite ; shuster 1 https://dialogcc.github.io/ 53.0 37.7 9.3 7.34 2.82 1.76 1.0 dialogcc (ours) mmdd mmdialog photochat tie 63.0 23.7 13.3 1) overall quality 2) image diversity 3) generalization # images/dialog # images/utterance 4.77 1.36 1.29 1.0 38.7 34 27.3 13.08 12.02 10.88 6.39 10.31 6.52 6.56 3.64 next response prediction image retrieval rc@1 avg. num win rate (%) head-to-head comparison figure 1: comparing dialogcc (ours) to three existing multi-modal dialogue dataset in terms of a quality, diversity, and generalization. rc@1 denotes the averaged contributed r@1 performance. @xcite @xcite @xcite . in practical situations, humans actively share images during conversations rather than merely talking about a given image, which is called image-sharing behavior @xcite . recent studies for the image-sharing have proposed multi-modal dialogue datasets, which are constructed through the crowd-sourcing (pho-tochat @xcite ), image-text similarity with human efforts (mmdd @xcite ), or social media platform @xcite ). however, existing multi-modal dialogue datasets have three significant limitations; (1) quality. recent studies have shown that a high-quality dataset enhances both the efficacy and the quality of the model training @xcite that is so sad i hope your pet gets better my dog got very sick and lay in bed for a couple weeks. i was so concerned. that is so sweet, i think that is why she got better, your caring ways she did get better eventually. during that time i tried to tend to her as much as i could. ## dialogcc in this section, we propose dialogcc, a highquality and diverse multi-modal social dialogue dataset. in order to construct dialogcc, we introduce an automatic pipeline, which consists of three steps: (1) collecting, (2) aligning, and (3) filtering. besides, we conduct a comprehensive analysis of our dataset with respect to quality and diversity by comparing three existing datasets, mmdd @xcite , photochat @xcite , and mmdialog @xcite . the overall pipeline is illustrated in figure 2 . in the following part of this section, we provide details about our proposed pipeline. ## conclusion in this paper, we propose the automatic pipeline for creating a multi-modal social dialogue dataset that involves aligning and filtering with gpt-4 and clip, respectively. we also propose a large-scale and high-quality multi-modal dialogue dataset, di-alogcc, which is constructed by leveraging the automatic pipeline with five text-only dialogue datasets and an image-text pair cc3m dataset. in a comprehensive analysis, compared to existing datasets mmdd, photochat, mmdialog, using di-alogcc helps achieve better quality in terms of various metrics. moreover, our dataset consists of many and various images per dialogue that can be beneficial in model generalization performance. extensive experiments demonstrate that a model trained with dialogcc increase model's robustness.