M oqa GPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model (2023)

Abstract: Multi-modal open-domain question answering typically requires evidence retrieval from databases across diverse modalities, such as images, tables, passages, etc. Even Large Language Models (LLMs) like GPT-4 fall short in this task. To enable LLMs to tackle the task in a zero-shot manner, we introduce MoqaGPT, a straightforward and flexible framework. Using a divide-and-conquer strategy that bypasses intricate multi-modality ranking, our framework can accommodate new modalities and seamlessly transition to new models for the task. Built upon LLMs, MoqaGPT retrieves and extracts answers from each modality separately, then fuses this multi-modal information using LLMs to produce a final answer. Our methodology boosts performance on the MMCoQA dataset, improving F1 by +37.91 points and EM by +34.07 points over the supervised baseline. On the MultiModalQA dataset, MoqaGPT surpasses the zero-shot baseline, improving F1 by 9.5 points and EM by 10.1 points, and significantly closes the gap with supervised methods. Our codebase is available at https://github.com/lezhang7/MOQAGPT.

## introduction large language models (llms) including chat-gpt (openai, 2022b), llama @xcite , palm2 @xcite , and the recently developed gpt4 (openai, 2022c), have fundamentally transformed the manner in which humans interact with machines. due to their vast knowledge repositories and chain-of-thought reasoning capability @xcite , these models have proven to be capable of providing answers to a broad range of questions across domains, without the need for training on specific tasks. nevertheless, these models face two significant challenges. first is the issue of hallucination @xcite , attributable to the fact that llms store their knowledge in their parameters. hallucination can seriously hamper 1 our codebase is available at https://github.com/ lezhang7/moqagpt . where was the movie the shape of water made? question [ consider an example outlined in fig. 1 where the question asked is, "where was the movie 'the shape of water' made?". to effectively answer this question, a human would employ a divideand-conquer approach. this strategy involves first retrieving relevant documents such as the movie poster, news reports, and table records for the shape of water. the individual would then derive the answer from the obtained references that might include terms like canada, toronto, or london, hertfordshire in england. comprehensive rea- table encoder image encoder text encoder modality detection multi-modality question answer extra information extract & encode special module large multi-modal open-domain qa annotations large multi-modal alignment dataset special annotations ( graph, ner, relation, etc. ) question type annotations soning would be applied to all potential answers, for instance, recognizing that toronto is related to canada. given this relationship, and the lack of strong ties between london, hertfordshire in england and other candidate answers, toronto is deemed the most probable answer and is selected as the final response. however, many existing models rely on a joint strategy @xcite @xcite where they attempt to retrieve and rank all modalities by training a joint embedding space. this approach, despite its application, has several shortcomings. firstly, the joint strategy lacks flexibility, necessitating retraining when new models and modalities are introduced. secondly, it poses considerable difficulties to train a joint embedding space and rank references encompassing more than two modalities. although some divide-and-conquer models have been proposed @xcite , they also come with limitations. these models require training a question type classifier, followed by the training of various question-answering models. each of these stages requires a significant amount of annotations, thus presenting a considerable challenge in their development and implementation. to enable llms to solve this task in a zero-shot manner, we propose multi-modal open-domain question answering gpt (moqagpt). mo-qagpt utilizes a divide-and-conquer approach and employs robust models to extract answers from various modalities. it further leverages llms as a reasoning mechanism, applying in-context learning to process the extracted information and generate the final response. compared with the traditional supervised methods, our framework, as depicted in fig. 2 , has three main advantages: flexibility: the moqagpt operates in zero-shot mode without relying on joint representation or inference, which allows easy replacement of individual modules with more advanced ones as they become available. furthermore, it can accommodate a wider range of modalities and eliminates the need to curate a multimodal open-domain dataset for training. trustworthiness: the framework's responses are based on the retrieved results. consequently, each answer can be traced back to its original source, thus making the model more trustworthy and reducing the risk of hallucination. interpretability: all intermediate outputs, including the retrieval results, candidate answers, and the final reasoning for answer synthesis, are produced in natural language, rendering the process of answering open-domain questions transparent and interpretable. we corroborate these advantages by presenting experimental results on two multi-modal opendomain question answering (mmoqa) datasets: mmcoqa @xcite and multmodalqa @xcite . both datasets require questions to be answered based on information retrieved from text, image, and table references. we conducted experiments using several of the latest models and demonstrated that our method is effective across all of them, highlighting our framework's flexibility. to demonstrate the trustworthiness of our method, we compared our outputs to those produced by directly querying llms. our outputs are less prone to hallucination, making them more trustworthy. lastly, we examined several success and failure cases of our method. thanks to the interpretable nature of our framework, we could identify the sources of errors. overall, our method exhibits robust zero-shot performance on both datasets, underscoring its promising potential. our contributions in this paper are threefold: (1) we propose moqagpt, a simple and effective framework, which is the first to enable llms to tackle multi-modal open-domain queries in a zeroshot setting. (2) we conduct extensive experiments involving multiple llms and vision-language models (vlms), thus validating the effectiveness of our approach. (3) we present empirical evidence that llms are capable of efficiently addressing mmoqa tasks when paired with other modalities. furthermore, we demonstrate that replacing each module with its superior version enhances performance, establishing this as a foundational framework for future zero-shot question-answering systems. earlier works in this field focus on model training. these include methodologies for joint embedding of multiple modalities (e.g. mae @xcite and manymodelqa @xcite ), the structured knowledge and unified retrievalgeneration based method skurg @xcite , and the multimodal graph transformer @xcite , which employs a graphbased quasi-attention mechanism for integrating multi-modal graph information. to the best of our knowledge, we are the first to introduce a zero-shot method for multi-modal open-domain question answering, marking a significant contribution. llm-based modular systems the development of modular neural networks, rooted in biology and neuroscience, can be traced back to the 1990s @xcite . before the rise of llms, modular neural networks like @xcite aimed to handle compositional tasks. they did this by decomposing them into sub-tasks, utilizing off-the-shelf language parsers, and then learning specialized neural modules for each. however, their applicability was limited, being constrained by parser performance and the need for hand-specified module types. the emergence of llms has renewed interest in this area. llms address the parsing challenges without necessitating additional training. consequently, this led to the proposition of various llmbased systems targeting an array of compositional reasoning challenges. examples include: toolformer @xcite , which trains language models to select tools. visual chatgpt @xcite , hugginggpt @xcite , and chameleon @xcite , all utilizing gpt to deduce tool sequences for response generation. vipergpt @xcite and visprog @xcite , leveraging codex and gpt3 @xcite respectively to produce python programs for visual reasoning tasks. yet, these methods don't address mmoqa. mmoqa inherently involves multiple steps, making it apt for a modular approach. our framework, therefore, capitalizes on llms to integrate and reason about information retrieved from various modalities for mmoqa. our approach is distinct as it requires no training, setting it apart from prior works like @xcite . it also differentiates itself from research such as @xcite ; @xcite ; @xcite with its emphasis on retrieval-based question answering. although chameleon @xcite supports both retrieval and question answering, it doesn't address the mmoqa tasks, especially those needing cross-modal information integration and reasoning. moreover, our method operates in a zeroshot setting, avoiding the need for intermediate programs, unlike chameleon which requires a fewshot python intermediate program. ## moqagpt moqagpt presents a general approach to generate answers to queries using a multi-modal knowledge base collection c, which encompasses text c txt , tables c tab , and images c img . the divideand-conquer strategy is accomplished through a two-stage process. first, the multi-modal question answer extraction stage ( §3.1) extracts answer candidates from stage1: zero-shot multimodal question answer extraction question stage2: zero-shot answers infusion @xmath0, a_text, a_tab , a_gpt] @xmath1, please select the best answer from the following candidates {answer_candidates}: vlms llms text &table qa gpts direct qa rule-based strategy ## experiments & results this section is organized as follows. in §4.1, we outline mmoqa datasets, metrics, and baselines. we then discuss retrieval performance in §4.2, question answering in §4.3, and mmoqa results in §4.4. lastly, we present an ablation study in §4.5 followed by a detailed case study in §4.6. ## conclusion in this study, we introduce the first zero-shot multimodal open-domain question answering framework, moqagpt, which enables llms to perform the mmoqa task. this framework is flexible, accommodating new models and modalities without requiring additional training. it stands out for its trustworthiness, being grounded on reliable retrieval results, and its interpretability, which is ensured by transparent intermediate outcomes. by leveraging llms and vlms, our model surpasses supervised methods in mmcoqa performance and significantly narrows the gap between zero-shot and supervised methods in multi-hop multimodal qa datasets. furthermore, our results indicate that models without robust knowledge storage capabilities, such as vicuna, are less suited for this task. we hope that our approach offers some insights and servers as a general and promising framework for multi-modal open-domain question answering.