Unifying Text, Tables, and Images for Multimodal Question Answering (2023)

Abstract: Multimodal question answering (MMQA), which aims to derive the answer from multiple knowledge modalities (e.g., text, tables, and images), has received increasing attention due to its board applications. Current approaches to MMQA often rely on single-modal or bi-modal QA models, which limits their ability to effectively integrate information across all modalities and leverage the power of pre-trained language models. To address these limitations, we propose a novel framework called UniMMQA, which unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. Additionally, we enhance cross-modal reasoning by incorporating a multimodal rationale generator, which produces textual descriptions of cross-modal relations for adaptation into the text-to-text generation process. Experimental results on three MMQA benchmark datasets show the superiority of UniMMQA in both supervised and unsupervised settings.

## introduction in typical question answering (qa) systems, answers are typically derived from a single modality, such as an image @xcite , passage @xcite , or table @xcite , without the need for cross-modality reasoning. however, in real-world scenarios, individuals often rely on multimodal information and reasoning from diverse knowledge sources to arrive at answers. multimodal question answering (mmqa) @xcite demands qa systems to perform reasoning across multiple knowledge modalities, including images, structured tables, textual passages, etc. preliminary attempts in mmqa @xcite typically utilize separate qa systems for different knowledge modal- * corresponding author. ## unimmqa framework given a question q and corresponding knowledge context @xmath0, t, p }, where i represents the image, t represents the table and p represents textual passage, unimmqa aims to generate an appropriate answer a. the overview of unimmqa is illustrated in figure 2 . ## conclusion in this paper, we present a novel mmqa framework called unimmqa, which unifies text, tables and images into a text-to-text format for plms. specifically, we first utilize an effective decoding strategy to generate diversified and informative image captions from the visual data. and we linearize the tabular data into textual sequence with table linearization methods. new york yankees. the yankees have won 27 world series championships. their most recent one came in 2009, under manager joe girardi, when they defeated the philadelphia phillies in six games. among mlb teams that are currently affiliated with the international league, when did the team with a baseball bat on their logo last win a world series tournament? rationale: new york yankees have a baseball cap on their logo. position tokens. meanwhile, a cross-modal rationale generator is adapted to produce explicit textual descriptions about the interrelations across different modalities. our experiments on three mmqa benchmark datasets show that unimmqa outperforms many mmqa methods and effectively take advantage of plms.