MT ab VQA : Evaluating Multi-Tabular Reasoning of Language Models in Visual Space (2025)

Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities in interpreting visual layouts and text. However, a significant challenge remains in their ability to interpret robustly and reason over multi-tabular data presented as images, a common occurrence in real-world scenarios like web pages and digital documents. Existing benchmarks typically address single tables or non-visual data (text/structured). This leaves a critical gap: they don’t assess the ability to parse diverse table images, correlate information across them, and perform multi-hop reasoning on the combined visual data. To bridge this evaluation gap, we introduce MTabVQA, a novel benchmark specifically designed for multi-tabular visual question answering. MTabVQA comprises 3,745 complex question-answer pairs that necessitate multi-hop reasoning across several visually rendered table images. We provide extensive benchmark results for state-of-the-art VLMs on MTabVQA, revealing significant performance limitations. We further investigate post-training techniques to enhance these reasoning abilities and release MTabVQA-Instruct, a large-scale instruction-tuning dataset. Our experiments show that fine-tuning VLMs with MTabVQA-Instruct substantially improves their performance on visual multi-tabular reasoning. Code and dataset are available online: .

## introduction in recent years, vision language models (vlms) and multimodal systems have demonstrated remarkable capabilities in interpreting complex visual layouts and text @xcite , enabling tasks ranging from document understanding @xcite , visual information extraction @xcite , and structured data qa @xcite to interactive processes like autonomous web navigation @xcite ). 1 project page: https://anshulsc.github.io/mtabvqa/ figure 1 : mtabvqa benchmark, illustrative example showing three tables (customers, reviews, services), a question requiring multi-table reasoning, the reasoning steps involved, and the final answer derived by a vision-language model. yet, as these models become increasingly capable of acting as visual agents that can browse screen data and execute complex instructions, a new challenge has emerged: the interpretation and reasoning over multi-tabular data presented as images @xcite . this challenge is particularly relevant in real-world scenarios, where tables often appear as images on web pages or digital documents. for instance, with approximately 24% of pdf documents being imageonly formats @xcite , their tabular data is inaccessible to traditional text-based models. extracting actionable insights from such sources may require a model to reference multiple table images simultaneously. traditional benchmarks @xcite @xcite in table understanding and question answering have primarily focused on single-table scenarios, often relying on textual or html representations @xcite . however, such benchmarks are unable to evaluate model performance on visually complex, multi-tabular data, which requires interpreting layout and structure beyond simple text or html. in many practical applications, such as financial analysis, e-commerce, and scientific research @xcite , key information is distributed across several tables, each with distinct layouts and visual structures. current benchmarks @xcite @xcite , rooted in single-table, non-visual formats (like text/html or relational databases), fail to assess critical capabilities: (1) understanding diverse visual table layouts presented as images, (2) parsing and correlating information across multiple, physically separate tables, and (3) executing multi-hop reasoning grounded in visual data. to bridge this gap, we propose multi-tabular visual question answering (mtabvqa), a novel benchmark specifically designed for assessing the visual reasoning capabilities of models on multitabular data represented as images. distinct from prior benchmarks that primarily focus on single tables @xcite @xcite or utilize non-visual (textual, structured) formats for multi-table reasoning @xcite @xcite , mtabvqa uniquely evaluates the integration of information across multiple tables. our benchmark, comprising 3,745 question-answer pairs, challenges models with complex queries across 14 distinct reasoning categories. these pairs are designed to necessitate multi-hop reasoning (e.g., involving aggregation, comparison, or fact-checking) by integrating information from two to five table images. mtabvqa enables a targeted evaluation of how well current models handle the process of extracting information from multiple table images and performing the multi-hop reasoning necessary to synthesize answers. our main contributions are: ## related work research in table understanding @xcite and multimodal reasoning @xcite has made significant advancements. initial efforts often centered on converting tables into text-based representations, such as markdown or html @xcite , enabling text-only language models to process them. while effective in controlled environments, this approach encounters limitations in real-world settings where tables frequently appear only as images within documents or web interfaces. processing visually rendered tables through multi-stage text-conversion pipelines @xcite presents inherent limitations. the pipelines are often complex and susceptible to ocr errors; they also tend to discard essential visual layout cues (e.g., merged cells, alignment), and risk compounding inaccuracies across stages. this highlights a critical need for models capable of interpreting and reasoning over tables directly from pixel data. moreover, most systems rely on ocr combined with llms, which makes them more prone to errors than developing a single unified model. our work focuses entirely on the challenge of extracting information directly from visual table data and drawing conclusions from it, addressing the complexities inherent in image-based table structures. ## mtabvqa dataset we introduce multi-tabular visual question answering (mtabvqa), a new dataset designed to evaluate and improve multi-hop reasoning over visually rendered tables. this dataset comprises two distinct and complementary components: mtabvqa, a benchmark with 3,745 qa pairs for evaluating model performance, and mtabvqa-instruct, a large-scale instruction-tuning dataset with 15,853 examples. the mtabvqa benchmark is further divided into four sub-datasets based on the primary source of the underlying table data. the detailed composition of both the sub-datasets and their sources is shown in ## experiments this section details the experiments conducted to evaluate vlm capabilities on visual multi-tabular reasoning using our mtabvqa benchmark. our experiments encompass three key areas: 1. benchmarking current vlms: we first establish baseline performance by evaluating leading open-source and proprietary vlms on the mtab-vqa split and compare it with our fine-tuned model (section 4.1). 2. evaluating post-training strategies: we investigate and compare the effectiveness of three distinct post-training techniques to improve vlm performance: chain-of-thought (cot) prompting, group relative policy optimization (grpo), and supervised fine-tuning (sft) @xcite (section 4.2).