M ^{2} - T ab F act: Multi-Document Multi-Modal Fact Verification with Visual and Textual Representations of Tabular Data (2025)

Abstract: Tabular data is used to store information in many real-world systems ranging from finance to healthcare. However, such structured data is often communicated to humans in visually interpretable formats (e.g. charts and textual paragraphs), making it imperative that fact-checking models should be able to reason over multiple pieces of structured evidence presented across different modalities. In this paper, we propose Multi-Document Multi-Modal Table-based Fact Verification (M ^{2} -TabFact), a challenging fact verification task that requires jointly reasoning over visual and textual representations of structured data. We design an automatic data generation pipeline that converts existing tabular data into descriptive visual and textual evidence. We then use Large Language Models to generate complex claims that depend on multi-document, multi-modal evidence. In total, we create 8,856 pairs of complex claims and multi-modal evidence through this procedure and systematically evaluate M ^{2} -TabFact with a set of strong vision-language models (VLM). We find that existing VLMs have large gaps in fact verification performance compared to humans. Moreover, we find that they are imbalanced when it comes to their ability to handle reason about different modalities, and currently struggle to reason about information extracted from multiple documents.

## introduction structured data are widely used to organize and present information in various settings ranging from web pages to spreadsheets and infographics. in an age where misinformation and hallucinated text generation continue to spread rapidly on the internet @xcite , building autonomous systems that can verify factual claims against structured data will lead to the reduction of misinformation and a safer experience on the internet. recently, several benchmarks have been proposed to evaluate automatic fact verification systems' ability to reason over structured data @xcite . however, two main limitations still remain. first, structured data in documents are commonly presented in complex but interpretable formats (e.g., visual chart plots or natural language summaries) rather than simple tables. second, existing table-based fact-checking systems built for these benchmarks simply assume that all of the evidence is contained in a single document or source. this is different from real-world scenarios, where human fact-checkers typically need to review multiple structured data evidence sources to evaluate the truthfulness of a complex claim. to address the above limitations, we propose multi-document multi-modal table-based fact-checking (m 2 -tabfact), a benchmark task which requires table-based fact verification systems to reason about information from multiple sources of structured data represented in multiple modalities. an example instance from our corpus is given in figure 1 . given a text hypothesis, the task is to verify the truthfulness of this claim against a chart and a text paragraph converted from two associated structured tables. solving this task entails decomposing the claim into two simpler pieces of information to retrieve, identifying the evidence from each modality necessary to retrieve the corresponding information, and then finally combining the information obtained from the two pieces of evidence to make a final decision on whether the claim is factual. m 2 -tabfact is constructed through an automatic pipeline involving four high-level steps. (1) evidence table collection: we split a table into two sub-tables to construct two plausibly related source tables for multi-document, multi-modal evidence creation. (2) multi-hop claim creation: we sample various data points from each source table and generate multi-hop claims that require multiple rea-the production of motor engines in the united kingdom experienced some fluctuations. in this period, the maximum monthly number of motor engines produced is 227624.0 achieved in june. three months later, the number of motor engines produced was 2,707 less than the maximum number. ## conclusion we introduce m 2 -tabfact, the first multidocument multi-modal fact-checking dataset over structured data to simulate the complex real-world fact-checking scenario on multi-source table evidence. we evaluate sota models including chartfocused vlms and powerful foundational vlms in a fine-tuned setting and a zero-shot setting. our best baseline achieves 59.2 % accuracy, which is still lagging far behind human's performance. we identify the major bottom neck of existing vlms'low performance on this dataset is the unbalanced capability to handle structured data in different modalities. we hope our research can inspire the development of robust fact-checking system against various structured data representations.