RITT : A Retrieval-Assisted Framework with Image and Text Table Representations for Table Question Answering (2025)

Abstract: Tables can be represented either as text or as images. Previous works on table question answering (TQA) typically rely on only one representation, neglecting the potential benefits of combining both. In this work, we explore integrating textual and visual table representations using multi-modal large language models (MLLMs) for TQA. Specifically, we propose RITT, a retrieval-assisted framework that first identifies the most relevant part of a table for a given question, then dynamically selects the optimal table representations based on the question type. Experiments demonstrate that our framework significantly outperforms the baseline MLLMs by an average of 13 Exact Match and surpasses two text-only state-of-the-art TQA methods on four TQA benchmarks, highlighting the benefits of leveraging both textual and visual table representations.

## introduction previous approaches in table question answering (tqa) represent tables as either textual sequences @xcite @xcite or as images @xcite , and process them by large language models (llms) or multi-modal large language models (mllms) accordingly. however, in real-life scenarios, tables often exist in both forms (e.g., html tables), or one form can be easily converted to the other via optical character recognition (ocr) or html rendering. this leads to increased interest in approaches that leverage both visual and textual table representations @xcite @xcite . current approaches using both representations either fine-tune an existing mllm using preference data collected by prompting mllms with different table representations of a tqa problem @xcite , or leverage instance-level features (e.g., table size) to determine the best representation for an mllm to process @xcite . the former requires careful data collection and training @xcite , while the latter struggles to effectively handle large tables due to the inherent limitations of current mllms @xcite . this work builds upon the latter approach, focusing on designing a training-free framework that can be easily applied across different datasets. we adopt the core idea proposed in fres @xcite to select the most suitable table representation of a tqa problem based on its question type. questions are classified as either retrieval questions, which only require locating information to be solved, or reasoning questions, which require both retrieval and reasoning. however, unlike fres, we introduce a novel a sub-table retriever that selects the most relevant part of a table to reduce input size. the module produces relevant table texts and images, which can be combined with the original full table and passed to an mllm for reasoning. to determine the optimal representation combinations for an mllm, we extend the analysis from @xcite to explore combinatorial scenarios, such as pairing retrieved table images with original textual representations and vice versa. our results indicate that combining textual and visual representations yields the best performance for reasoning questions, while textual representation alone is sufficient for retrieval questions. based on these findings, we propose ritt, a training-free retrieval-assisted framework leveraging image and text representations of tables. it comprises four modules: a sub-table retriever, a question classifier, a table reformatter, and an mllm reasoner. experimental results show that ritt outperforms baseline mllms by an average of 13 exact match (em) points, and surpasses two state-of-the-art text-only tqa systems, demonstrating the clear benefits of leveraging both table representations. lastly, we provide an ablation study highlighting the contribution of each component. table representations for tqa. most prior work processes tables as texts @xcite or as images @xcite . @xcite fine-tune an existing mllm using preference data collected by prompting an mllm with different table representations. @xcite propose a rule-based framework fres to select the best table representation for an mllm. they obtain the rules by comparing different representations under varying scenarios controlled by table size and question type. their findings indicate that different representations perform differently under varying conditions. for instance, passing large tables in textual format to llms can lead to better performance than passing large tables in images to mllms. though the textual format is more robust than the visual format when handling larger tables, it still faces challenges with large tables. in this work, we propose a sub-table retriever to mitigate the impact of table size on representation selection. moreover, we extend existing analyses to cover combinatorial cases where retrieved sub-tables are combined with original tables. ## framework figure 2 shows an overview of our proposed system ritt. it contains four parts: a sub-table retriever that filters for the most relevant cells, a question classifier to determine a question type, a table reformatter to reformat a retrieved sub-table based on question type, and a table reasoner mllm to output an answer to a given tqa problem. ## experiments datasets. we evaluate ritt with the test sets of three aforementioned tqa benchmarks used during our analysis in section 3.2: wtq, tabfact (small test), and hitab. we also include one additional dataset, wikisql @xcite , that has not been used in our analysis to test the generalizability of our method. models and baselines. our framework contains an mllm table reasoner and an llm retriever/classifier. 3 for mllms, we choose the best performing mllm from our previous analysis: pixtral-12b (see a.3 for individual model's performance) as well as a fine-tuned mllm for tqa: tablellava-7b @xcite . we use qwen-2-72b as the llm backbone in our framework. as baselines, we choose the backbone mllm without applying ritt. in addition, we compare ritt with two sota frameworks that use only table text representations: tablerag @xcite and graphotter @xcite mllm (qwen-2-vl-72b). 4 as a result, all compared frameworks use the same backbone model. ## results and discussions results. ## conclusions in this paper, we explored leveraging both textual and visual table representations using mllms for tqa. to handle the challenges of large table inputs and representation selection, we proposed ritt, a retrieval-assisted framework that retrieves the most relevant sub-table, classifies the question type, and dynamically determines the optimal representations to an mllm reasoner based on the question type. extensive experiments on four tqa benchmarks demonstrated the advantages of our framework over baseline mllms as well as frameworks utilizing only textual representations. ablation studies further confirmed the effectiveness of each proposed component. our findings highlight the benefits and promising potential of integrating both table representations for tqa.