MMD ialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2023)

Abstract: Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog dataset to facilitate multi-modal conversation better. MMDialog is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 88x. Second, it contains massive topics to generalize the open domain. To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios. In addition, we build two baselines for the above tasks with state-of-the-art techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset is available in https://github.com/victorsungo/MMDialog .

## introduction empowering machines to converse like humans is a long-cherished goal of ai community, and there is growing interest in developing open-domain conversational agents @xcite @xcite @xcite . to usher machines into the real visual physical world of human, it is a desirable trait of conversational agents to understand, perceive, and respond appropriately to multi-modality contexts beyond pure text @xcite @xcite at the base of a muddy ditch is the first primrose of my spring -glowing in the grey, a little spot of hope, brave, beautiful and perfect. hi chris, wow well spotted with the beautiful flower, i love walking alongside the river where there is a bluebell way love every photo. especially the weeping willow. thanks. it's nice to enjoy the wildlife nature and walk all the way to winchester's great scenery. i live in scotland. we have woods opposite with bluebells but not as thick as yours, but have a river with kingfisher, heron and dipper. i would love to visit scotland and witness the beautiful hills seas, and wildlife nature. we have a lots of cheeky squirrels and wood pigeons, i love to listen them in morning. that's an amazing story thanks for sharing. et al., 2020; @xcite , which is similar to communicating through messenger tools (e.g., facebook, whatsapp, and wechat) in reality. yet existing approaches to building multi-modal dialogue systems are primarily data-driven, requiring the collection of a large-scale dataset first. to facilitate this line of research, the community emerges a few dialogue datasets incorporating visual information @xcite ## dialogue creation mmdialog is a large-scale multi-turn dialogue dataset towards multi-modal open-domain conversations. it derives from a worldwide social media platform where users can converse with each other and share daily life experiences freely through various modalities, including plain text, photos, or even videos. we design the data collection process into three phases: in the first phase, we extensively manually collect the hashtags commonly used by users and cover as many domains as possible. the second phase starts from the seed hashtags collected in phase one and focuses on collecting all turns that contain at least one image, which we subsequently refer to as anchors. for each anchor, we retrieve all the turns that replied to it and the turn it replied to. we could obtain a complete dialogue session by performing the above process iteratively. in the final phase, we also elaborately design a series of data filtering and post-processing steps to eliminate invalid cases and improve the quality of multi-modal dialogues in mmdialog. to protect the privacy and security of the data, users, and platform, mmdialog is only available to academic researchers under strict terms. ## evaluation of multi-modal responses 6.1 general metrics in task-1, we follow @xcite and evaluate the quality of the generated responses by using metrics from both text and visual modalities. for the text modality, we compare the generated text to the ground-truth responses using metrics like bleu @xcite and rouge @xcite . for the visual modality, we measure the quality of the generated images using metrics like inception score @xcite (is) and clip score @xcite (clip-sim). suppose the model does not generate a corresponding textual or visual element, we assign it a score of zero in that case, indicating no relevance between the "empty" element and the ground truth. however, we could not directly adopt the same strategy for perplexity (ppl, for text) and frechet inception distance @xcite (fid, for image). as a lower ppl (or fid) implies a closer distance between generated text (or image) distribution and textual dialogue response generator clip-t clip-v transformer 𝒖 𝟏 𝒕 , 𝒖 𝟐 𝒗 , 𝒖 𝟑 𝒗 , 𝒖 𝑲 𝒕 … … 𝒖 𝟐 𝒄 , 𝒖 𝟑 𝒄 , … … image-to-description description-to-image 𝒓 𝟏 𝒄 , … … 𝒓 𝒋-𝟏 𝒄 𝒓 𝒋 𝒕 𝒓 𝟏 𝒗 , … … 𝒖 𝟏 𝒕 , 𝒖 𝟐 𝒗 , 𝒖 𝑲 𝒕 , … 𝒓 𝒋-𝟏 𝒗 , … … [𝑼, ෩ 𝑹 <𝒋 ] real-world text (or image) distribution. unfortunately, we could not assign an infinity value (similar with zero value for bleu/is metrics) standing for "empty" element for both metrics. in task-2, we could also compute the recall@n scores for text and image in similar way. specifically, for each retrieval step, if the predicted modality intent of next element mismatches with the ground-truth, we could also assign the recall score of this step to zero values. the final recall scores of an example is the average scores of all elements group by the same modality. inspired by @xcite which use f1 score to evaluate intent prediction, we extend it from single to multi elements: our f1 score could measure whether the models predict right order of elements modality in a response. specifically, we first get the modality sequence of generated/retrieved and ground-truth responses as @xmath0. then, the f1 score is formulated as: ## experiments experiments are conducted on mmdialog to assess two baselines. we perform response/intent predictions for all turns except the first turn of each dialogue and consider all previous turns as context. ## conclusion we presented mmdialog, a large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. by extracting turns associated with images and their surrounding contexts from more than 4k topics, mmdialog provides a diverse and open-domain dataset. to facilitate research on building a more engaging multi-modal dialogue system, we define multi-modal response generation and retrieval tasks, and the mm-relevance metric based on mmdialog. we also build baseline models and conduct comprehensive performance analyses. we believe mmdialog can serve as a rich resource to propel research in the multi-modal conversation, for years to help the community propose better methods suited to more scenarios.