FM 2 DS : Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answering (2025)

Abstract: Multimodal multihop question answering (MMQA) requires reasoning over images and text from multiple sources, an essential task for many real-world applications. Despite advances in visual question answering, this multihop setting remains underexplored due to a lack of quality datasets. Existing methods focus on single-hop, single-modality, or short texts, limiting real-world applications like interpreting educational documents with long, multimodal content. To fill this gap, we introduce FM2DS, the first framework for creating a high-quality dataset for MMQA. Our approach consists of a 5-stage pipeline that involves acquiring relevant multimodal documents from Wikipedia, synthetically generating high-level questions and answers, and validating them through rigorous criteria to ensure data quality. We evaluate our methodology by training models on our synthesized dataset and testing on two benchmarks: MultimodalQA and WebQA. Our results demonstrate that, with an equal sample size, models trained on our synthesized data outperform those trained on human-collected data by 1.9 in exact match (EM) score on average. Additionally, we introduce M2QA-Bench with 1k samples, the first benchmark for MMQA on long documents, generated using FM2DS and refined by human annotators.

## introduction multimodal multihop question answering (mmqa) involves answering complex questions by integrating information from text, images, and tables. in real-world applications such as interpreting medical documents, this challenge is amplified by the need to reason over long, multimodal content. current methodologies in mmqa typically leverage in-context learning capability existing datasets (e.g webqa) fm 2 ds (ours) relevant docs question generation answer generation query generation • large human effort: most documents are linked by human annotators • limited context: only relevant information snippets are provided figure 1 : unlike traditional datasets that rely on human annotators, templates, or snippets, fm 2 ds is fully automated, using long documents as sources and applying validation to ensure questions are answerable, multimodal, and multihop. in this work, we propose fm 2 ds, a novel data synthesis framework designed specifically for mmqa over long documents. our approach synthesizes mmqa data from documents that are interconnected through various relationships, such as thematic similarities or sequential events. this framework leverages naturally occurring document relationships and requires minimal hand-crafted data, thereby broadening the range of reasoning types used in question generation. as illustrated in figure 1 , fm 2 ds enables the generation of non-templated question-answer pairs based on full documents rather than brief information snippets. the data generated by our method fm 2 ds includes query component -a step-bystep guide for retrieving relevant information from multiple documents -enabling smaller models trained on this synthesized data to learn how to tackle complex questions in a manner similar to larger models. this methodology allows users to create a custom mmqa dataset with fewer than ten human-annotated samples, thereby facilitating the fine-tuning of smaller lvlms for specific applications. fm 2 ds leverages wikipedia's extensive knowledge base and hyperlink structure to select document pairs with shared topical relevance or hyperlink connections and prompt lvlms to perform question generation, question answering, and query generation. we incorporate validation steps to enhance the quality of the generated data and discard any outputs that are factually incorrect. through empirical evaluation on established mmqa benchmarks, we show that fm 2 ds significantly improves model performance, achieving on average a 1.9 exact match (em) score improvement across two benchmarks: multimodalqa and webqa. our key contributions are: (i) introducing a new framework for synthesizing high-quality mmqa training data for lvlms; (ii) using a robust validation pipeline to ensure data quality; (iii) introducing a challenging mmqa benchmark requiring reasoning over multiple modalities and sources; and (iv) showing that models fine-tuned on our synthetic data outperform those trained on humanlabeled datasets, advancing mmqa while reducing manual effort. ## related work within the question answering (qa) literature, synthesis of training data has been predominantly focused on unimodal (text-only) scenarios. we review various similar works that have established the foundation for our work in few-shot data synthesis. unimodal data synthesis synthetic data is increasingly used for model training. @xcite show that combining labeled and synthetic text from language models (lms) improves nlp performance. entire synthetic datasets have also been created for tasks like classification @xcite , with @xcite demonstrating gpt-3.5's effectiveness in generating reliable classification data. similarly, @xcite show that synthetic data can significantly boost small models on multihop qa with minimal human annotation. ## experiments and results this section compares our synthesized dataset to human-annotated ones. all experiments used one in-context example during synthesis (see appendix d for effects of varying number in-context examples). gpt-4o was used in the pipeline (appendix e shows results with other lvlms). models were evaluated using exact match (em) for accuracy and f1 for partial match quality. further experimental details can be found in appendix b. ## human evaluation of answer validation to evaluate the accuracy and impact of our automatic answer validation component, we conducted a human evaluation study. after generating questions for 100 samples, we continued the pipeline using two methods: fm 2 ds with and without answer validation. these samples were divided into four batches of 25, with each batch evaluated by three participants to mitigate potential bias or human errors. twelve participants used our evaluation platform (appendix h) to judge answer correctness. fm2ds with answer val win 32.4% loss 1 % draw 64.2% both incorrect 2.4% human evaluation of the impact of answer validation step on reducing factual errors in answer generation participants were instructed to verify the correctness of each answer by reviewing the associated wikipedia pages. once they determined the accuracy of each answer, they were asked to select one of four options: (1) method 1's answer is correct, (2) method 2's answer is correct, (3) both methods generated the correct answer, or (4) neither answer is correct (method 1 and method 2 were randomly assigned to the datasets generated with and without answer validation, respectively.). the results, shown in figure 5 , reveal a clear trend: answer validation increases the likelihood of correct answers, even though the model without validation still produced correct answers in over 60% of cases. the fleiss' kappa was 0.91, indicating strong agreement, though this is expected, as the task involved factual questions with definitive answers rather than subjective judgments. ## conclusion & future works we present fm 2 ds, a novel methodology for synthesizing high-quality data for multimodal multihop question answering. unlike existing approaches that are limited to single-hop and singlemodality settings, fm 2 ds generates complex qa pairs that require reasoning over multiple modalities and sources, with minimal human intervention. our framework enables the creation of a large-scale dataset that significantly boosts model performance, surpassing models trained on humancurated data in terms of test accuracy. these results demonstrate the effectiveness of synthetic data for advancing the state of the art in multimodal multihop qa. additionally, fm 2 ds offers a scalable and efficient solution for training datahungry language models. for future work, we plan to synthesize mmqa samples using sources beyond wikipedia, including multilingual content, code snippets, videos, and other diverse information type.