MAGID : An Automated Pipeline for Generating Synthetic Multi-modal Datasets (2024)

Abstract: Development of multimodal interactive systems is hindered by the lack of rich, multimodal (text, images) conversational data, which is needed in large quantities for LLMs. Previous approaches augment textual dialogues with retrieved images, posing privacy, diversity, and quality constraints. In this work, we introduce M ultimodal A ugmented G enerative I mages D ialogues (MAGID), a framework to augment text-only dialogues with diverse and high-quality images . Subsequently, a diffusion model is applied to craft corresponding images, ensuring alignment with the identified text. Finally, MAGID incorporates an innovative feedback loop between an image description generation module (textual LLM) and image quality modules (addressing aesthetics, image-text matching, and safety), that work in tandem to generate high-quality and multi-modal dialogues. We compare MAGID to other SOTA baselines on three dialogue datasets, using automated and human evaluation. Our results show that MAGID is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small.

## introduction in recent years, advancements in large language models (llms) have expanded possibilities and research directions in ai, with studies highlighting their extensive capabilities in handling dialogue datasets @xcite . specifically, there is a growing interest in their application to multimodal dialogue datasets, given that sharing images is an integral aspect of human-human conversations @xcite @xcite . several multi-modal dialogue datasets like mmdialog @xcite , dialogcc @xcite , and photochat @xcite have been introduced for training multimodal llms. these datasets either use a retrieval-based approach, pulling images from set image banks, such as ms-coco @xcite , or restrict the dialogue to only one image per conversation, even if they involve real human-human chats. moreover, when leveraging real-world datasets from platforms like social media, issues related to privacy concerns and image quality become significant challenges for training. as a result, these methods limit the diversity of images since the small image database cannot adequately capture the wide range of real human-human conversations @xcite . additionally, they face challenges stemming from low-quality images containing harmful and private content @xcite and shortage of accessible data @xcite , particularly when utilizing real human-human conversations from social media sources. to address these challenges, we propose magid, a generative-based multi-modal dialogue creation framework. as illustrated in figure 1 , magid aims at converting existing text-only data into context-enriched multimodal data by addressing the two research challenges: (i) how to find the most suitable utterances that can be enhanced by adding images and (ii) how to generate realistic and diverse images that do not have harmful and private contents. you left your halloween costume at my place hey, did i leave anything at your house? i was worried it was lost forever. ## magid pipeline in transitioning from text-only to multi-modal dialogue, there exist two core challenges. the first is the identification of the most suitable utterances within the dialogue that can be enhanced by images. the second is the creation of corresponding, accurate images that align with the selected utterances. in this regard, we need to ensure a harmonious and coherent match between the image and the text, achieving acceptable image-text alignment. we have addressed these challenges through the implementation of the following three key modules in figure 1 , namely llm-based scanner, diffusion-based image generator, and quality assurance module, which are detailed in the subsequent sections. ## evaluation we scrutinize the efficacy and applicability of the multi-modal dataset generated by magid. here are three pivotal questions we addressed in evaluation: the first and third question delves into a quantitative analysis, probing the accuracy and quality of the data generated by magid. moreover, the second question is crucial, as a failure of magid to meet human evaluation standards would result in a low-quality training dataset that is unable to get positive human-centric assessments. in addition, in supplementary (section e), we have studied training multimodal model ## magid dataset as a proof of concept, and consistent with studies like @xcite , we employed text-only datasets such as dailydialog @xcite , persona-chat @xcite , and photochat @xcite (by replacing its images with magid) to generate a multi-modal dataset foot_1 of 58,279 dialogues. based on the results of our experiments, we used gpt-3.5 to transform 52,527 input dialogues and gpt-4 to augment the rest. ## conclusion we presented a generative, fully automated pipeline, madig, designed to transform textonly datasets into multi-modal variants, harnessing the power of llms through prompt engineering. this solution addresses limitations faced by preceding methods, notably in terms of data privacy, accessibility, constrained image distribution, and occurrences of unsuitable or non-consensual content. crucially, our pipeline permits the substitution of real, potentially privacy-compromising images with synthetic counterparts. we thoroughly evaluated magid using human assessment, quantitative analyses with various llms, and an in-depth ablation study. the promising results highlight generative ai's capability to stand as an alternative to traditional data generation methods, like mechanical turk. looking ahead, our dataset paves the way for developing large multi-modal language models that can engage with users via both text and visuals.