OMG - QA : Building Open-Domain Multi-Modal Generative Question Answering Systems (2024)

Abstract: We introduce OMG-QA, a new resource for question answering that is designed to evaluate the effectiveness of question answering systems that perform retrieval augmented generation (RAG) in scenarios that demand reasoning on multi-modal, multi-document contexts. These systems, given a user query, must retrieve relevant contexts from the web, which may include non-textual information, and then reason and synthesize these contents to generate a detailed, coherent answer. Unlike existing open-domain QA datasets, OMG-QA requires systems to navigate and integrate diverse modalities and a broad pool of information sources, making it uniquely challenging. We conduct a thorough evaluation and analysis of a diverse set of QA systems, featuring various retrieval frameworks, document retrievers, document indexing approaches, evidence retrieval methods, and LLMs tasked with both information retrieval and generation. Our findings reveal significant limitations in existing approaches using RAG or LLM agents to address open questions that require long-form answers supported by multi-modal evidence. We believe that OMG-QA will be a valuable resource for developing QA systems that are better equipped to handle open-domain, multi-modal information-seeking tasks.

## introduction modern question answering systems are explored within two primary frameworks. the first framework operates under the premise of a limited context, providing all necessary information to answer queries. this approach, which treats qa as a reading comprehension exercise, assesses the system's ability to extract and interpret information from a given context to formulate responses @xcite @xcite @xcite . although this method offers a detailed examination of the systems' comprehension and reasoning skills, it relies on the availability of chosen context, limiting its applicability in many real-world scenarios where the context to address the question is not directly available. the second framework, also known as open-domain qa, addresses this limitation by requiring the system to source information from large-scale knowledge sources -such as text corpora, databases or the internet -in response to any user query @xcite @xcite @xcite . typically, these systems utilize a two-stage design: a retrieval stage that efficiently identifies broadly relevant contexts from extensive knowledge sources, and a subsequent reading stage that mirrors the closed setting. with advancements in large language models (llms), these systems have primarily benefited the reading stage, demonstrating enhanced proficiency in interpreting and reasoning with the retrieved content. enhancing open-domain qa systems presents two primary challenges. the first challenge involves enhancing retrieval stage using llms while ensuring efficiency and scalability. to tackle this, several studies have integrated llms into retrieval frameworks through methods like query expansion, ranking adjustments @xcite @xcite , or embedding extraction for dense retrieval @xcite @xcite @xcite @xcite . the second challenge is enabling qa systems to retrieve and interpret multimodal content, such as tables, images, and videos. research efforts to address this have included creating a unified embedding space that allows for the retrieval and ranking of context across different modalities @xcite @xcite @xcite @xcite . although there have been many attempts to ad-dress these challenges, there remains a notable gap: the lack of a comprehensive benchmark capturing the complexities of real-world tasks and can effectively evaluate these advancements. in response, we introduce open-domain multi-modal generative question answering dataset foot_0 (omg-qa). unlike existing open-domain multi-modal qa datasets @xcite @xcite that primarily feature factoid questions @xcite and call for concise, single noun-phrase or entity-based answers, omg-qa challenges qa systems to retrieve and reason across content in various modalities within an open setting, ultimately resulting in the generation of detailed narratives or explanations. additionally, we implement various types of llm systems, described in section 3, which are evaluated by our dataset to assess their ability to retrieve multi-modal content in an open setting. ## omg-qa we define open-domain multi-modal generative question answering as the task of producing a longform answer a, which is a structured discourse that presents entities and their relationships in response to a question q. this process is based on a large-scale knowledge source k, from which the system must retrieve multiple pieces of evidence e 1 , e 2 , . . . , e n to substantiate the answer. to ensure that the systems generate answers grounded in the retrieved evidence, we also mandate systems to explicitly cite the evidences used within the answer. an example of our dataset is provided in figure 1 . ## qa systems constructing an open-domain qa system requires three fundamental components: an index, a retriever, and an answerer. for omg-qa, we assume that all qa systems in our study first retrieve documents from wikipedia using established search apis. furthermore, we aim to assess the llm systems' capabilities of retrieving fine-grained content. to this end, we implement an evidence retrieval process in all our qa systems, which involves indexing fine-grained content within documents and deploying corresponding evidence retrievers. following these two retrieval steps, the system employs an answerer to aggregate, reason, and synthesize various pieces of evidence to produce the final answer. an illustration of our qa systems is provided in figure 1 . ## conclusion in this study, we introduce omg-qa, which challenges qa systems to retrieve and reason across text, tables, and images to generate long-form answers. our experiments reveal that multi-index setups outperform single index setting in evidence retrieval, dense retrievers excel over sparse and generative retrievers, and multi-round retrieval enhances document recall but not necessarily evidence relevance in all cases. the choice of llms has minimal impact on retrieval performance; however, the best retrieval configuration paired with gpt-4, equipped with memory on retrieval history and self-reflection, showed superior results in overall evaluations. these findings emphasize the importance of integrating multi-modal content and sophisticated retrieval strategies in developing more capable qa systems, positioning omg-qa as a robust benchmark for future advancements.