Table Understanding and (Multimodal) LLM s: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data (2025)

Abstract: Tables are among the most widely used tools for representing structured data in research, business, medicine, and education. Although LLMs demonstrate strong performance in downstream tasks, their efficiency in processing tabular data remains underexplored. In this paper, we investigate the effectiveness of both text-based and multimodal LLMs on table understanding tasks through a cross-domain and cross-modality evaluation. Specifically, we compare their performance on tables from scientific vs. non-scientific contexts and examine their robustness on tables represented as images vs. text. Additionally, we conduct an interpretability analysis to measure context usage and input relevance. We also introduce the TableEval benchmark, comprising 3017 tables from scholarly publications, Wikipedia, and financial reports, where each table is provided in five different formats: Image, Dictionary, HTML, XML, and LaTeX. Our findings indicate that while LLMs maintain robustness across table modalities, they face significant challenges when processing scientific tables.

## introduction tables are one of the most ubiquitous tools for presenting data in a structured or semi-structured manner. they are commonly represented in a variety of textual (e. g., html, l a t e x, xml) or image formats (e. g., png, jpeg) and used across domains such as finance, medicine, and business, as well as in research and education. in recent years, there has been a growing interest in table understanding (tu) techniques @xcite @xcite @xcite , aiming to extract and interpret information and knowledge contained in tables for tasks such as question answering (qa) and table-to-text generation (t2t) @xcite @xcite . while large language models (llms) demonstrate strong performance in a wide range of applications @xcite @xcite @xcite , their ability to understand (semi-)structured data remains under-researched @xcite -especially for tables from scientific sources such as peer-reviewed articles, conference proceedings, and pre-prints. 1 there is also limited research on the impact of the representation modality of structured data (i. e., image vs. text) on model performance @xcite , and to the best of our knowledge, there are no approaches yet that specifically address scientific tables. in particular, most tu studies primarily focus on tables from nonscientific contexts such as wikipedia @xcite @xcite @xcite . however, compared to these domains, scientific tables often include technical terminology, complex concepts, abbreviations, and dense numerical values, requiring domain-specific knowledge and strong arithmetic reasoning skills @xcite . recent works @xcite indicate that scientific tables present challenges to multimodal llms (mllms) and incorporating such (semi-)structured data into pretraining improves performance. as the number of published articles continues to increase rapidly @xcite @xcite , tu for scientific contexts, e. g., for scholarly document processing including information extraction and research knowledge graph construction, is becoming even more relevant. finally, we notice that interpretability analysis @xcite for tu has received little attention and remains underexplored @xcite . in this paper, we address the aforementioned gaps by examining the efficiency of both llms and mllms on a set of tu tasks. specifically, we compare their ability to handle (semi-)structured data from scientific and non-scientific sources and explore the effects of image vs. diverse text-based table representations on model performance. we also conduct feature importance analyses to interpret the use of context information in llms. figure 1 illustrates the main phases of our experiments. our contributions can be summarised as follows: ## tableeval benchmark since no existing dataset covers both scientific and non-scientific tables across text and image modalities, we construct a benchmark tailored to our evaluation. this section outlines the collection processes of data ( §2.1) and diverse table formats ( §2.2). ## experiments we benchmark various (m)llms using individual data subsets and representations of tables from tableeval. this is followed by an interpretability analysis applied to the output yielding attributions from a gradient-based method. in the following, we first describe the experimental set up ( §3.1), then report and analyse the results ( §3.2). ## related work earlier tu studies leverage llms by representing tables as sequential text, either through naïve linearisation or by incorporating delimiters and special tokens @xcite . some works focus on fine-tuning llms to enhance tu @xcite @xcite @xcite , while others explore llms' table reasoning abilities through prompt engineering @xcite @xcite . however, compared to natural language, tables present unique challenges to llms due to their varying layout structures, feature heterogeneity, and a large number of components leading to excessively long sequences @xcite . the latter is particularly problematic, as most llms become inefficient due to the quadratic complexity of self-attention @xcite . with recent advances in vision and multimodality research, using mllms for tu has gained increasing attention with models like gpt-4 (openai et al., 2024) and gemini @xcite , being widely adopted. although, similar to llms, mllms also struggle with understanding structured data @xcite . several studies examine the impact of the table representation on models' efficiency, indicating that different table formats suit specific tu tasks and llms at hand @xcite @xcite while these studies offer insights into the effectiveness of (m)llms in interpreting structured data across formats, they focus primarily on nonscientific contexts like wikipedia and finance. this is likely due to the abundance of established, largescale datasets based on tables from these sources, including wikitables @xcite , totto @xcite , and tabfact, @xcite , to name a few. furthermore, interpretability for tu tasks remains under-researched, as related works mainly consider unstructured text and are disconnected from downstream applications @xcite , rarely focusing on other long-form tasks like retrieval-augmented generation @xcite or qa @xcite . @xcite use attributions to make tabular qa explainable but they are constrained to the text-to-sql setup. unlike prior studies, this paper focuses on crossdomain and cross-modality evaluation, comparing the performance and explanations of (m)llms on both scientific and non-scientific tables, covering image and diverse text representations of tables. ## conclusion we conducted an evaluation study to explore the robustness of diverse (m)llms on scientific vs. non-scientific tables across image and four text formats. the findings reveal that current models obtain decent performance across both vision and text modalities but significantly struggle with scientific tabular data. additionally, we explored the applicability of interpretability methods to tu tasks to get insights into the decision-making of llms. we found feature attributions to be a useful tool for revealing model uncertainty, its attention to table structure and relevant content, and tokenisation differences which might potentially affect predictions.