Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (2024)

Abstract: Existing datasets for tabular question answering typically focus exclusively on text within cells. However, real-world data is inherently multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content in tables. With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data. This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We explore their ability to reason on tables that integrate both images and text, introducing MMTabQA, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs, understanding visual context, and comparing visual content across images. These findings establish our dataset as a robust benchmark for advancing AI’s comprehension and capabilities in analyzing multimodal structured data.

## introduction tables are crucial for efficiently summarizing and conveying information across various fields. in real-world applications, they often include images representing entities, such as team logos in sports scoreboards and product features in e-commerce tables (fig. 1 ). in medicine, tables may display visual symptoms for comparing diseases, while educational tables might include molecular diagrams or images of plant species. wikipedia tables frequently incorporate images, such as team logos in sports articles or comparative tables for scientists, nobel laureates, and ship classes. political party tables often feature election symbols and charts illustrating seat wins. this integration of images enriches the data's depth and informativeness. q1 how to unlock the phone which has a dual horizontal camera? a1 fingerprint scanner q2 which phone combine three camera lens with latest processor? a2 iphone 11 pro or iphone 11 pro max q3 which phone comes with the fewest color options? a3 iphone 8 or 8+ table comparing iphone features understanding and interpreting these multimodal tables is crucial across various domains. in healthcare, they help doctors compare disease symptoms for accurate diagnosis and treatment planning. in education, students use visual aids in tables to better understand complex concepts. in e-commerce, consumers rely on product comparison tables to make informed purchasing decisions. advances in modeling techniques, including table pre-training and targeted fine-tuning, have greatly improved reasoning capabilities for semistructured tables @xcite . furthermore, large language models (llms) have shown remarkable performance across diverse domains, achieving state-of-the-art performance on various tabular reasoning tasks @xcite @xcite . despite extensive exploration of inference and reasoning over tables most prior works @xcite have primarily focused on text-only tables. thus, developing advanced ai models to process multimodal tables is essential. these models must integrate textual and visual data for comprehensive analysis across fields. improving these models can enhance the accuracy and efficiency of tasks in healthcare diagnostics, education, and consumer decision-making, thereby enriching data presentation and enhancing its informativeness and utility. reasoning with multimodal tables poses significant challenges. table reasoning, as illustrated in fig. 1 , necessitates entity disambiguation within the table context. for instance, disambiguating the a13 square as representing the latest a13 processor chip is essential for answering q2. additionally, visual reasoning over individual images in the table is crucial, such as using the phone images to determine the camera alignment for answering q1. similarly for q3, understanding phone colors through their visual representation and counting them could be challenging. to answer these questions well, model must understand the images in the table and its relation to other cells (images and text), which involves complex reasoning such as visual analysis, numerical interpretation, temporal sequencing, and entity relationship identification. however, the lack of datasets to evaluate vision-language models on multimodal table reasoning has left this area largely unexplored. therefore, this paper aims to investigate the research question: can current vision-language models handle complex reasoning in multimodal tables? to effectively tackle this challenge, we introduce a new task called knowledge-aware reasoning over multimodal semi-structured tables. due to the timeconsuming and costly process of curating a new human-annotated dataset on multimodal tables, we repurpose existing wikipedia datasets into a multimodal format. our framework replaces recognizable entities in textual wikipedia tables with their representative images, creating the multimodal table question answering (mmtabqa) dataset repurposed using four wikipedia tables-based question-answering datasets. in mmtabqa, we categorize questions into explicit (mentioning an image-replaced entity explicitly), answer-mention (referencing an imagereplaced entity in the answer), and implicit (involving image-replaced entities in intermediate reasoning). we also generate synthetic visual questions by enhancing explicit questions with visual attributes of the mentioned entity and validate them through human evaluation. evaluating various state-of-theart closed and open-source llms and vlms using diverse modeling approaches on mmtabqa reveals challenges in entity disambiguation, understanding table structures, and performing visual reasoning. we aim for our dataset to be a robust benchmark for evaluating vision-language models (vlms) on complex multimodal tabular reasoning. we summarize our contributions as below: the dataset, along with associated scripts, are available at https://mmtabqa.github.io/ . ## multimodal tabular reasoning today's vlms face multiple challenges when reasoning with multimodal tabular questionanswering datasets. below, we describe these challenges in detail: ## modelling strategies to benchmark the model performance on our mmtabqa dataset, we define four baselines: ## result and analysis we observe that our models' performance varies significantly across different datasets, methods, and models. we provide a detailed analysis of these variations below: ## comparison with related work recent advancements in natural language processing (nlp) have expanded beyond traditional homogeneous tables to incorporate additional modalities. works such as @xcite @xcite integrate paragraph context alongside tables for enhanced tabular question answering. meanwhile, efforts like @xcite introduce images alongside text and tables, yet they do not address the non-homogeneous modalities found in mmtabqa tables, containing text-only tables. visual table understanding has also gained attention, with approaches such as @xcite converting textual tables into vi-sual formats for multimodal reasoning. however, these efforts do not capture the multimodal complexities inherent in mmtabqa. datasets like @xcite focus on general knowledge for visual question answering, while others @xcite @xcite emphasize finegrained entity knowledge within images. multimodal entity linking and disambiguation across different modalities are explored in @xcite , echoing the entity linking challenges posed by mmtabqa. in visual question answering, advancements have been made in handling multiple images @xcite , though it remains less explored compared to single-image tasks. unlike prior work, which typically focuses on either textual or visual elements separately, our task confronts the novel challenge of multimodal tables that integrate multiple texts and multiple images within table cells. this involves addressing explicit, implicit, and answer-mention questions, while also advancing visual understanding within the framework of table-based reasoning. ## conclusion this research explores whether nlp models can effectively reason with knowledge on multimodal structured data. we investigate their ability to process tables that combine images and text, introducing mmtabqa, a new dataset for this task. our experiments reveal substantial challenges for ai models in integrating and interpreting multiple text and image inputs, understanding visual context, and comparing visual content across images. our findings position mmtabqa as a crucial benchmark for advancing ai's capabilities in analyzing multimodal structured data.