Synthetic Multimodal Question Generation (2024)

Abstract: Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. A key challenge with evaluating MMRAG is the paucity of high-quality datasets matching the question styles and modalities of interest. In light of this, we propose SMMQG, a synthetic data generation framework. SMMQG leverages interplay between a retriever, large language model (LLM) and large multimodal model (LMM) to generate question and answer pairs directly from multimodal documents, with the questions conforming to specified styles and modalities. We use SMMQG to generate an MMRAG dataset of 1024 questions over Wikipedia documents and evaluate state-of-the-art models using it, revealing insights into model performance that are attainable only through style- and modality-specific evaluation data. Next, we measure the quality of data produced by SMMQG via a human study. We find that the quality of SMMQG-generated synthetic data is on par with the quality of the crowdsourced benchmark MMQA and that downstream evaluation results using both datasets strongly concur.

## introduction following the increased adoption of retrieval augmented generation (rag) @xcite for text-based question-answering (qa), there has been much interest in extending rag to the multimodal setting @xcite @xcite . in multimodal rag (mmrag), qa is performed by a large language model (llm) or large multimodal model (lmm) grounded in sources that span modalities such as text, tables and images. as with rag, mmrag has the potential to increase answer quality and transparency when compared with † work done while at c3 ai. closed-book qa, where models directly answer questions without access to external knowledge. a major challenge encountered when implementing mmrag systems is evaluation, which is typically done using fixed benchmark datasets. these datasets consist of (source(s), question, answer) tuples that enable separate evaluation of the retriever and the qa model. some representative datasets include mmqa @xcite , mmmu @xcite and webqa @xcite . the problem with this approach is that we cannot tailor the evaluation questions to our desired specifications. we identify two aspects of the evaluation questions we may wish to control. the first aspect is the question style. the style of a question determines the reasoning abilities required to answer it, and certain models perform better on certain question styles than others. examples of question styles include mathematical @xcite , multi-hop @xcite and extractive @xcite question styles. the modality of a question refers to the modality or modalities (e.g. text, table, image) of the sources required to answer it, and both retrieval @xcite and qa @xcite performance depends on the modalities of their inputs. given the sensitivity of mmrag performance to question styles and modalities, we want our evaluation questions to match the styles and modalities of the questions our system is likely to encounter. such a benchmark may not exist, making it difficult to perform the comprehensive evaluations needed to reveal important model deficiencies. to address this issue, we introduce a method for synthetic multimodal question generation we call smmqg. smmqg is a synthetic data generation framework that leverages interplay between a retriever, an llm and an lmm with incontext learning @xcite to generate multimodal questions and answers based directly on input documents. crucially, smmqg enables fine-grained control over the styles and modalities of questions, and is capable of producing both unimodal and cross-modal questions. in order to demonstrate the utility of smmqg, we first use it to create a dataset of 1024 questions and answers of various styles and modalities from wikipedia documents. we then use this dataset to unearth novel style-and modality-specific insights into the mmrag performance of various models. such insights would be difficult to obtain without styleand modality-specific evaluation data, which sm-mqg can produce in an automatic and scalable fashion. one concern with synthetic data generation is that the resulting data is of low quality. to assuage this concern and verify that smmqg produces high quality data, we conduct a human study to measure the quality of our synthetic dataset. we find that our dataset's quality is on par with or better than that of popular crowdsourced benchmark mmqa @xcite when measured along five different metrics. we also show that downstream evaluation results obtained using our smmqg dataset display strong concurrence @xcite with those obtained using mmqa, demonstrating that our synthetic dataset can be used in place of mmqa for model selection. ## conclusion we introduce smmqg, a framework for generating synthetic multimodal questions and answers grounded in multimodal documents that adhere to user-specified question styles and modalities. we demonstrate that an smmqg-generated evaluation dataset is able to reveal novel style-and modalityspecific insights into the performance of state-ofthe-art retrievers, llms and lmms. through a human study and by measuring dataset concurrence, we also show that the quality of data generated by smmqg is on-par with the quality of data from crowdsourced benchmark mmqa and that our dataset can be used in place of mmqa for model selection. we hope that smmqg will enable automatic, large-scale and tailored evaluation of mmrag systems, thereby facilitating their adoption in practical applications.