W iki M ix QA : A Multimodal Benchmark for Question Answering over Tables and Charts (2025)

Abstract: Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). While vision-language large models (VLLMs) have demonstrated improvements across various tasks, their effectiveness in processing long-context vision inputs remains unclear. This paper introduces WikiMixQA, a benchmark comprising 1,000 multiple-choice questions (MCQs) designed to evaluate cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages spanning seven distinct topics. Unlike existing benchmarks, WikiMixQA emphasizes complex reasoning by requiring models to synthesize information from multiple modalities. We evaluate 12 state-of-the-art vision-language models, revealing that while proprietary models achieve ~70% accuracy when provided with direct context, their performance deteriorates significantly when retrieval from long documents is required. Among these, GPT-4-o is the only model exceeding 50% accuracy in this setting, whereas open-source models perform considerably worse, with a maximum accuracy of 27%. These findings underscore the challenges of long-context, multi-modal reasoning and establish WikiMixQA as a crucial benchmark for advancing document understanding research.

## introduction documents serve as a fundamental medium for preserving and exchanging information, with millions being generated daily across various domains. beyond plain text, they often include complex layouts, tables, and images, making automatic document understanding (du) a crucial challenge. nlp-driven du enables efficient extraction, organization, and interpretation of this information, supporting realworld information retrieval and decision-making. 1 code and dataset is released here. figure 1 : an example from wikimixqa illustrating a question whose answer relies on the information presented in the accompanying table and chart. a notable challenge in du arises from the prevalence of documents containing large tables and charts, which can be difficult for humans to process and analyze. a question-answering (qa) system would help humans get insights from documents with such interjections easily. over the past few years, several visual question answering (vqa) benchmarks have been developed to assess the du capabilities of vision-language large models (vllms) across various aspects, including handling tables, charts, and document layouts. however, most existing benchmarks primarily focus on single-page documents. another key challenge in automatic du is the ability to answer complex questions that require integrating information across multiple sections of a document and reasoning over different modalities. current benchmarks largely lack multi-hop questions where models must synthesize information from multiple modalities-such as text, tables, and charts-to derive correct answers @xcite . furthermore, (2) for each article, we identify table-chart pairs that exhibit semantic similarity. (3) we employ gpt-4-turbo to generate multiple-choice questions (mcqs) based on each table-chart pair. (4) human annotators assess and validate the quality of the generated questions to ensure accuracy and relevance. to bridge this gap, we introduce wikimixqa, a benchmark dataset constructed using wikipedia as the primary source of documents. wikimixqa focuses on long, digital-only documents that contain multiple large tables and charts. the questions in our dataset are specifically designed to require multi-modal reasoning over these structured elements, covering a broad spectrum of topics. the dataset comprises 1,000 multiple-choice questions (mcqs) derived from ∼4k wikipedia pages, with each document averaging 24.18 pages and 1815.01 ± 2825.16 textual tokens. the questions span diverse domains, including economy, geography, history, politics, science, sport, and wikimedia. to ensure the need for multi-modal reasoning, questions are structured to require information from either two tables ( we conduct an extensive evaluation on wikimixqa using four open-source and eight closed-source state-of-the-art vllms across three different experimental settings, where we vary the level of contextual information provided to the models. the results, summarized in ## dataset construction this section outlines the pipeline employed for collecting wikipedia documents, extracting and selecting their associated modalities (i.e., tables and charts), and generating the multiple-choice question (mcq) samples that constitute wikimixqa. an overview of the wikimixqa creation pipeline is illustrated in 2, with a detailed explanation provided in the appendix b. ## analysis in the oracle setting, where models receive direct access to relevant context, proprietary models perform significantly better. gpt-4o achieves the highest accuracy, with claude and gemini models closely following. in contrast, open-source models exhibit poor performance, performing at or near random levels. for the wikidoc setting, we exclude closedsource models due to their limited context length, which prevents them from processing the full wikipedia snapshots required for answering questions. among open-source models, only gpt-4o surpasses 50% accuracy. this observation suggests that while these models perform well when provided with explicitly relevant information, they struggle when required to process long-context inputs. the challenge arises from their need to first locate relevant information within extensive text in our human evaluation, annotators achieved an accuracy of 87% in the oracle setting, revealing a substantial performance gap of approximately 17% between current vllms and human performance. this discrepancy underscores the challenges that lvlms face in document understanding and highlights the necessity of our benchmark for advancing research in this area. ## related work visual question answering (vqa) is a crucial subtask of document understanding (du), where the objective is to generate natural language answers to questions based on a given visual document. previous research has explored the du capabilities of vision large language models (vllms) by introducing new datasets and benchmarks. many of these datasets are designed to evaluate specific components, such as tables @xcite or charts @xcite @xcite @xcite , and are often constrained to single-page documents. while recent benchmarks have attempted to extend document vqa beyond single-page settings, they still face limitations in terms of cross-page reasoning, domain diversity, and question complexity. for example, mp-docvqa @xcite , an extension of docvqa @xcite , does not include cross-page questions. dude @xcite introduces a small proportion of cross-page questions but is limited by its reliance on crowd-sourced annotations, which often result in less challenging and rigorous questions, many of which focus on document layout rather than deeper content understanding. similarly, slide-vqa @xcite incorporates crosspage questions but is tailored to slide decks, which typically contain lower information density compared to other document types. doc2soargraph @xcite includes a few multi-page queries but remains limited to pdfs as its data format. financebench @xcite addresses some of these challenges by including long-context documents and practical cross-page questions. benchmark sources origin # docs. # questions cross-page # tokens answer type evidence source mmlongbench-doc multi digital + scan 135 1k ✓ 21214.1 abst + ext. t, l, f, ch, m dude multi digital + scan 5k 41k ✓ 1,831.53 abst. + ext. t, l, f, ch, m mp-docvqa industry docs mostly scans 6k 50k ✗ 2026.6 ext. t, l, f, ch visualmrc web pages digital 10k 30k ✗ 154.19 abst. t, l, f, ch infographicsvqa infographics digital 5.4k 30k ✗ 287.98 abst + ext. t, l, f, ch, m tat-dqa finance reports digital 2.7k 16k ✗ 576.99 abst. + ext. t, l wikimixqa wikipedia digital ∼ 4k 1k ✓ 1815.01 abst. + ext. t, l, f, ch, m however, its exclusive focus on financial reports and open-ended answer formats requires expertlevel manual evaluation, restricting its applicability to broader domains. similarly, crab @xcite focuses on question-answering based on events spanning multiple documents, however, its scope is limited to the domain of causal understanding. more recently, mmlongbench-doc @xcite was introduced, incorporating questions from diverse sources, with approximately one-third being cross-page questions. despite this, the dataset is derived from only 130 documents, limiting its domain coverage and diversity. a more detailed comparison of existing datasets is presented in ## conclusion in this paper, we introduce wikimixqa, a multimodal visual question-answering (vqa) benchmark designed to assess the long-context document understanding (du) capabilities of vision-language large models (vllms). our benchmark consists of 1,000 multiple-choice questions (mcqs) that necessitate complex, multi-hop reasoning over visual data, including charts and tables. we conducted an extensive evaluation of both closed-source and open-source models. our results indicate that while closed-source models perform well when provided with precisely relevant information, their performance degrades significantly in settings where they must process and extract relevant details from long-context visual data. in contrast, state-of-the-art open-source models exhibit performance close to random, suggesting fundamental challenges in their reasoning capabilities. these findings highlight that current vllms struggle with vqa tasks requiring the extraction and integration of dispersed information from ex-tended contexts. we hope that wikimixqa serves as a valuable resource for the research community in identifying and addressing the limitations of vllms in reasoning and long-context visual understanding.